duckdb/tutorial/08-numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f0e8cc2-b968-4059-b48a-a99bad1feecb",
   "metadata": {},
   "source": [
    "# Tutorial 8: Running NumPy In Your Data Warehouse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8790d140-229f-48fb-8986-9aa77c7b4210",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\"> <b>Before we get started: </b> \n",
    "    <ul style=\"list-style-type: none;margin: 0;padding: 0;\">\n",
    "        <li>✍️ To run this notebook, you need to have Ponder installed and set up on your machine. If you have not done so already, please refer to our <a href=\"https://docs.ponder.io/getting_started/quickstart.html\">Quickstart guide</a> to get started.</li> \n",
    "        <li>📁 This tutorial makes use of the <code>ponder.db</code> database that we created in <a href=\"https://github.com/ponder-org/ponder-notebooks/blob/main/duckdb/tutorial/01-getting-started.ipynb\">Tutorial #1</a>. You can also download the file <a href=\"https://github.com/ponder-org/ponder-datasets/raw/main/ponder.db\">here</a>.</li> \n",
    "        <li>📖 Otherwise, if you're just interested in browsing through the tutorial, keep reading below!</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28eea415-26b6-423e-b8de-2e37c01a0184",
   "metadata": {},
   "source": [
    "Pandas and NumPy are the two most popular Python data science libraries, used by more than 60% of Python developers. NumPy supports linear algebra operations in Python, and as a result, is the fundamental building block of machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e5aa6b-259a-4845-baf8-05dea286d2c1",
   "metadata": {},
   "source": [
    "Ponder lets you run NumPy commands directly in your warehouse. This means you can work with the NumPy API to build data and ML pipelines, and let DuckDB take care of scaling and security for you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cba60d7b-8399-4574-baf1-1b4c8def6ff4",
   "metadata": {},
   "source": [
    "Here, we'll show a few examples of Ponder in action with NumPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5bf11dc6-686c-451f-9341-27ad1193e5da",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-05-12 10:39:07 - Creating session qRHh27oJQYoTJOR3vhcxzRn_G3_3MtGiwumj3MFCqI\n"
     ]
    }
   ],
   "source": [
    "import ponder; ponder.init()\n",
    "import modin.pandas as pd\n",
    "import duckdb\n",
    "duckdb_con = duckdb.connect(\"../ponder.db\")\n",
    "ponder.configure(default_connection=duckdb_con)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5ac5c435-f1e8-4d34-aa95-c3167b17fbe7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_sql(\"PONDER_CUSTOMER\", duckdb_con)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f61b127-fd79-4ac9-980a-84bfd03506ef",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\"> <b>Note: </b> <span>NumPy support is currently part of Modin's experimental API, please drop us a note at <a href\"mailto:support@ponder.io\">support@ponder.io</a> if you run into any issues. Feedback welcome!</span></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "05a81146-43a0-4ab4-b1b6-1db3ae221cf3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import modin.config as cfg\n",
    "cfg.ExperimentalNumPyAPI.put(True)\n",
    "import modin.numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1152fa4f-dbdb-4cb1-88c1-ff3cdf1d7163",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: Using Modin's new NumPy API. To convert from a Modin object to a NumPy array, either turn off the ExperimentalNumPyAPI flag, or use `modin.utils.to_numpy`.\n"
     ]
    }
   ],
   "source": [
    "arr = df.select_dtypes(\"number\").astype(\"float\").to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b64eda-5ad1-4398-9861-891f25b714b7",
   "metadata": {},
   "source": [
    "We can convert the numerical values of the dataframe into Modin's NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e82312b4-e2a0-4412-9add-db8591d9fd3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "modin.numpy.arr.array"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(arr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4833cc0e-4aff-403e-a582-59f9fd8e3702",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6.00010e+04,  1.40000e+01,  9.95756e+03],\n",
       "       [ 6.00020e+04,  1.50000e+01,  7.42460e+02],\n",
       "       [ 6.00030e+04,  1.60000e+01,  2.52692e+03],\n",
       "       [ 6.00040e+04,  1.00000e+01,  7.97522e+03],\n",
       "       [ 6.00050e+04,  1.20000e+01,  2.50474e+03],\n",
       "       [ 6.00060e+04,  2.20000e+01,  9.05140e+03],\n",
       "       [ 6.00070e+04,  1.20000e+01,  6.01717e+03],\n",
       "       [ 6.00080e+04,  2.00000e+00,  5.62144e+03],\n",
       "       [ 6.00090e+04,  9.00000e+00,  9.54801e+03],\n",
       "       [ 6.00100e+04,  2.10000e+01,  3.49791e+03],\n",
       "       [ 6.00110e+04,  8.00000e+00,  3.42248e+03],\n",
       "       [ 6.00120e+04,  2.30000e+01,  6.71600e+02],\n",
       "       [ 6.00130e+04,  0.00000e+00, -4.85690e+02],\n",
       "       [ 6.00140e+04,  1.60000e+01,  7.93215e+03],\n",
       "       [ 6.00150e+04,  9.00000e+00,  4.62239e+03],\n",
       "       [ 6.00160e+04,  2.00000e+00,  4.48097e+03],\n",
       "       [ 6.00170e+04,  1.00000e+00,  3.65317e+03],\n",
       "       [ 6.00180e+04,  0.00000e+00,  5.75983e+03],\n",
       "       [ 6.00190e+04,  1.00000e+00,  3.44477e+03],\n",
       "       [ 6.00200e+04,  1.50000e+01,  1.18730e+03],\n",
       "       [ 6.00210e+04,  5.00000e+00,  9.91951e+03],\n",
       "       [ 6.00220e+04,  0.00000e+00, -7.59740e+02],\n",
       "       [ 6.00230e+04,  2.10000e+01, -1.29700e+01],\n",
       "       [ 6.00240e+04,  2.00000e+00, -6.64000e+00],\n",
       "       [ 6.00250e+04,  2.20000e+01,  7.29615e+03],\n",
       "       [ 6.00260e+04,  2.10000e+01,  7.50813e+03],\n",
       "       [ 6.00270e+04,  2.40000e+01,  6.11522e+03],\n",
       "       [ 6.00280e+04,  6.00000e+00,  3.49499e+03],\n",
       "       [ 6.00290e+04,  1.40000e+01,  9.28115e+03],\n",
       "       [ 6.00300e+04,  2.10000e+01,  6.39750e+03],\n",
       "       [ 6.00310e+04,  8.00000e+00,  6.76574e+03],\n",
       "       [ 6.00320e+04,  1.60000e+01,  2.89686e+03],\n",
       "       [ 6.00330e+04,  2.40000e+01, -4.93590e+02],\n",
       "       [ 6.00340e+04,  1.40000e+01,  7.23069e+03],\n",
       "       [ 6.00350e+04,  1.30000e+01,  7.18950e+03],\n",
       "       [ 6.00360e+04,  1.20000e+01,  7.89790e+03],\n",
       "       [ 6.00370e+04,  4.00000e+00,  1.03242e+03],\n",
       "       [ 6.00380e+04,  7.00000e+00,  2.77118e+03],\n",
       "       [ 6.00390e+04,  1.00000e+01, -7.65420e+02],\n",
       "       [ 6.00400e+04,  4.00000e+00,  3.44678e+03],\n",
       "       [ 6.00410e+04,  8.00000e+00,  1.94382e+03],\n",
       "       [ 6.00420e+04,  1.70000e+01,  8.68196e+03],\n",
       "       [ 6.00430e+04,  0.00000e+00,  7.37030e+02],\n",
       "       [ 6.00440e+04,  4.00000e+00, -8.12910e+02],\n",
       "       [ 6.00450e+04,  1.00000e+00,  8.38811e+03],\n",
       "       [ 6.00460e+04,  2.00000e+01,  1.02257e+03],\n",
       "       [ 6.00470e+04,  2.00000e+00,  2.26476e+03],\n",
       "       [ 6.00480e+04,  8.00000e+00,  1.79813e+03],\n",
       "       [ 6.00490e+04,  9.00000e+00,  2.12883e+03],\n",
       "       [ 6.00500e+04,  1.60000e+01,  5.18199e+03],\n",
       "       [ 6.00510e+04,  1.40000e+01,  4.73758e+03],\n",
       "       [ 6.00520e+04,  1.60000e+01,  4.51517e+03],\n",
       "       [ 6.00530e+04,  0.00000e+00,  6.51551e+03],\n",
       "       [ 6.00540e+04,  1.30000e+01, -8.61620e+02],\n",
       "       [ 6.00550e+04,  1.00000e+01,  3.72535e+03],\n",
       "       [ 6.00560e+04,  3.00000e+00,  8.88028e+03],\n",
       "       [ 6.00570e+04,  2.00000e+00,  9.84870e+02],\n",
       "       [ 6.00580e+04,  2.30000e+01,  6.67212e+03],\n",
       "       [ 6.00590e+04,  2.40000e+01,  2.33746e+03],\n",
       "       [ 6.00600e+04,  6.00000e+00,  2.63607e+03],\n",
       "       [ 6.00610e+04,  1.40000e+01,  5.34963e+03],\n",
       "       [ 6.00620e+04,  0.00000e+00,  6.21099e+03],\n",
       "       [ 6.00630e+04,  1.60000e+01,  7.19147e+03],\n",
       "       [ 6.00640e+04,  9.00000e+00,  8.02604e+03],\n",
       "       [ 6.00650e+04,  1.70000e+01,  9.75739e+03],\n",
       "       [ 6.00660e+04,  9.00000e+00,  2.47224e+03],\n",
       "       [ 6.00670e+04,  4.00000e+00,  1.87419e+03],\n",
       "       [ 6.00680e+04,  2.30000e+01,  4.49281e+03],\n",
       "       [ 6.00690e+04,  5.00000e+00,  4.65611e+03],\n",
       "       [ 6.00700e+04,  1.20000e+01, -7.26590e+02],\n",
       "       [ 6.00710e+04,  1.00000e+00,  7.05068e+03],\n",
       "       [ 6.00720e+04,  2.00000e+01,  6.79179e+03],\n",
       "       [ 6.00730e+04,  9.00000e+00,  7.31030e+02],\n",
       "       [ 6.00740e+04,  1.00000e+00,  8.36434e+03],\n",
       "       [ 6.00750e+04,  1.40000e+01,  8.59308e+03],\n",
       "       [ 6.00760e+04,  2.00000e+01,  3.04010e+03],\n",
       "       [ 6.00770e+04,  4.00000e+00,  9.00061e+03],\n",
       "       [ 6.00780e+04,  2.10000e+01, -7.04560e+02],\n",
       "       [ 6.00790e+04,  2.40000e+01,  3.32955e+03],\n",
       "       [ 6.00800e+04,  0.00000e+00,  6.89240e+02],\n",
       "       [ 6.00810e+04,  1.50000e+01, -9.24450e+02],\n",
       "       [ 6.00820e+04,  0.00000e+00,  3.64547e+03],\n",
       "       [ 6.00830e+04,  1.20000e+01,  8.43043e+03],\n",
       "       [ 6.00840e+04,  1.80000e+01,  8.25173e+03],\n",
       "       [ 6.00850e+04,  1.90000e+01,  7.95774e+03],\n",
       "       [ 6.00860e+04,  4.00000e+00,  5.83595e+03],\n",
       "       [ 6.00870e+04,  2.20000e+01,  6.81075e+03],\n",
       "       [ 6.00880e+04,  2.00000e+01,  3.12900e+02],\n",
       "       [ 6.00890e+04,  1.30000e+01,  9.84912e+03],\n",
       "       [ 6.00900e+04,  7.00000e+00,  4.29350e+03],\n",
       "       [ 6.00910e+04,  2.20000e+01,  9.91995e+03],\n",
       "       [ 6.00920e+04,  3.00000e+00,  6.71167e+03],\n",
       "       [ 6.00930e+04,  2.20000e+01,  7.01033e+03],\n",
       "       [ 6.00940e+04,  1.20000e+01, -4.08230e+02],\n",
       "       [ 6.00950e+04,  4.00000e+00,  8.10000e+02],\n",
       "       [ 6.00960e+04,  1.20000e+01,  4.62025e+03],\n",
       "       [ 6.00970e+04,  2.10000e+01,  1.62661e+03],\n",
       "       [ 6.00980e+04,  0.00000e+00,  1.44968e+03],\n",
       "       [ 6.00990e+04,  2.10000e+01,  8.76765e+03],\n",
       "       [ 6.01000e+04,  1.60000e+01, -5.85000e+00]], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efcbe12c-5bee-47ad-9150-66782d3a374b",
   "metadata": {},
   "source": [
    "We can perform reduce operations such as `np.sum` and `np.mean` across the entire matrix: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "acbdd7ee-5bef-48ee-84dd-b4dfea2b003a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6448157.578125"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42433f63-cdd7-451f-93d7-10eeb70a0939",
   "metadata": {},
   "source": [
    "or we can perform the reduce operation along a specific axis: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "54490f21-89cb-4dcb-a907-44d1937589b2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: Modin does not yet support broadcasting between nested 1D arrays and 2D arrays.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[23324.188],\n",
       "       [20253.154],\n",
       "       [20848.64 ],\n",
       "       [22663.072],\n",
       "       [20840.58 ],\n",
       "       [23026.467],\n",
       "       [22012.057],\n",
       "       [21877.146],\n",
       "       [23188.67 ],\n",
       "       [21176.303],\n",
       "       [21147.16 ],\n",
       "       [20235.533],\n",
       "       [19842.436],\n",
       "       [22654.049],\n",
       "       [21548.797],\n",
       "       [21499.656],\n",
       "       [21223.725],\n",
       "       [21925.943],\n",
       "       [21154.924],\n",
       "       [20407.434],\n",
       "       [23315.17 ],\n",
       "       [19754.088],\n",
       "       [20010.344],\n",
       "       [20006.453],\n",
       "       [22447.717],\n",
       "       [22518.377],\n",
       "       [22055.406],\n",
       "       [21176.33 ],\n",
       "       [23108.049],\n",
       "       [22149.5  ],\n",
       "       [22268.248],\n",
       "       [20981.62 ],\n",
       "       [19854.47 ],\n",
       "       [22426.229],\n",
       "       [22412.5  ],\n",
       "       [22648.633],\n",
       "       [20357.807],\n",
       "       [20938.727],\n",
       "       [19761.193],\n",
       "       [21163.594],\n",
       "       [20664.273],\n",
       "       [22913.654],\n",
       "       [20260.01 ],\n",
       "       [19745.03 ],\n",
       "       [22811.37 ],\n",
       "       [20362.857],\n",
       "       [20771.254],\n",
       "       [20618.043],\n",
       "       [20728.943],\n",
       "       [21749.33 ],\n",
       "       [21600.86 ],\n",
       "       [21527.725],\n",
       "       [22189.502],\n",
       "       [19735.127],\n",
       "       [21263.451],\n",
       "       [22979.76 ],\n",
       "       [20347.957],\n",
       "       [22251.04 ],\n",
       "       [20806.82 ],\n",
       "       [20900.69 ],\n",
       "       [21808.209],\n",
       "       [22090.998],\n",
       "       [22423.49 ],\n",
       "       [22699.68 ],\n",
       "       [23279.797],\n",
       "       [20849.08 ],\n",
       "       [20648.396],\n",
       "       [21527.936],\n",
       "       [21576.703],\n",
       "       [19785.137],\n",
       "       [22374.227],\n",
       "       [22294.596],\n",
       "       [20271.01 ],\n",
       "       [22813.115],\n",
       "       [22894.025],\n",
       "       [21045.367],\n",
       "       [23027.203],\n",
       "       [19798.146],\n",
       "       [21144.184],\n",
       "       [20256.412],\n",
       "       [19723.85 ],\n",
       "       [21242.49 ],\n",
       "       [22841.81 ],\n",
       "       [22784.578],\n",
       "       [22687.248],\n",
       "       [21975.318],\n",
       "       [22306.584],\n",
       "       [20140.299],\n",
       "       [23317.04 ],\n",
       "       [21463.5  ],\n",
       "       [23344.318],\n",
       "       [22268.89 ],\n",
       "       [22375.11 ],\n",
       "       [19899.256],\n",
       "       [20303.   ],\n",
       "       [21576.084],\n",
       "       [20581.537],\n",
       "       [20515.893],\n",
       "       [22962.549],\n",
       "       [20036.717]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean of every row returning object of same dimensions\n",
    "np.mean(arr, axis=-1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c9d47c-c37b-4720-bce0-d345524180a7",
   "metadata": {},
   "source": [
    "We can also do element-wise matrix operations such as addition of two matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "674ea11f-d770-42b4-aaae-c860361e3956",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6.9958562e+04, 2.8000000e+01, 6.9958562e+04],\n",
       "       [6.0744461e+04, 3.0000000e+01, 6.0744461e+04],\n",
       "       [6.2529922e+04, 3.2000000e+01, 6.2529922e+04],\n",
       "       [6.7979219e+04, 2.0000000e+01, 6.7979219e+04],\n",
       "       [6.2509738e+04, 2.4000000e+01, 6.2509738e+04],\n",
       "       [6.9057398e+04, 4.4000000e+01, 6.9057398e+04],\n",
       "       [6.6024172e+04, 2.4000000e+01, 6.6024172e+04],\n",
       "       [6.5629438e+04, 4.0000000e+00, 6.5629438e+04],\n",
       "       [6.9557008e+04, 1.8000000e+01, 6.9557008e+04],\n",
       "       [6.3507910e+04, 4.2000000e+01, 6.3507910e+04],\n",
       "       [6.3433480e+04, 1.6000000e+01, 6.3433480e+04],\n",
       "       [6.0683602e+04, 4.6000000e+01, 6.0683602e+04],\n",
       "       [5.9527309e+04, 0.0000000e+00, 5.9527309e+04],\n",
       "       [6.7946148e+04, 3.2000000e+01, 6.7946148e+04],\n",
       "       [6.4637391e+04, 1.8000000e+01, 6.4637391e+04],\n",
       "       [6.4496969e+04, 4.0000000e+00, 6.4496969e+04],\n",
       "       [6.3670172e+04, 2.0000000e+00, 6.3670172e+04],\n",
       "       [6.5777828e+04, 0.0000000e+00, 6.5777828e+04],\n",
       "       [6.3463770e+04, 2.0000000e+00, 6.3463770e+04],\n",
       "       [6.1207301e+04, 3.0000000e+01, 6.1207301e+04],\n",
       "       [6.9940508e+04, 1.0000000e+01, 6.9940508e+04],\n",
       "       [5.9262262e+04, 0.0000000e+00, 5.9262262e+04],\n",
       "       [6.0010031e+04, 4.2000000e+01, 6.0010031e+04],\n",
       "       [6.0017359e+04, 4.0000000e+00, 6.0017359e+04],\n",
       "       [6.7321148e+04, 4.4000000e+01, 6.7321148e+04],\n",
       "       [6.7534133e+04, 4.2000000e+01, 6.7534133e+04],\n",
       "       [6.6142219e+04, 4.8000000e+01, 6.6142219e+04],\n",
       "       [6.3522988e+04, 1.2000000e+01, 6.3522988e+04],\n",
       "       [6.9310148e+04, 2.8000000e+01, 6.9310148e+04],\n",
       "       [6.6427500e+04, 4.2000000e+01, 6.6427500e+04],\n",
       "       [6.6796742e+04, 1.6000000e+01, 6.6796742e+04],\n",
       "       [6.2928859e+04, 3.2000000e+01, 6.2928859e+04],\n",
       "       [5.9539410e+04, 4.8000000e+01, 5.9539410e+04],\n",
       "       [6.7264688e+04, 2.8000000e+01, 6.7264688e+04],\n",
       "       [6.7224500e+04, 2.6000000e+01, 6.7224500e+04],\n",
       "       [6.7933898e+04, 2.4000000e+01, 6.7933898e+04],\n",
       "       [6.1069422e+04, 8.0000000e+00, 6.1069422e+04],\n",
       "       [6.2809180e+04, 1.4000000e+01, 6.2809180e+04],\n",
       "       [5.9273578e+04, 2.0000000e+01, 5.9273578e+04],\n",
       "       [6.3486781e+04, 8.0000000e+00, 6.3486781e+04],\n",
       "       [6.1984820e+04, 1.6000000e+01, 6.1984820e+04],\n",
       "       [6.8723961e+04, 3.4000000e+01, 6.8723961e+04],\n",
       "       [6.0780031e+04, 0.0000000e+00, 6.0780031e+04],\n",
       "       [5.9231090e+04, 8.0000000e+00, 5.9231090e+04],\n",
       "       [6.8433109e+04, 2.0000000e+00, 6.8433109e+04],\n",
       "       [6.1068570e+04, 4.0000000e+01, 6.1068570e+04],\n",
       "       [6.2311762e+04, 4.0000000e+00, 6.2311762e+04],\n",
       "       [6.1846129e+04, 1.6000000e+01, 6.1846129e+04],\n",
       "       [6.2177828e+04, 1.8000000e+01, 6.2177828e+04],\n",
       "       [6.5231992e+04, 3.2000000e+01, 6.5231992e+04],\n",
       "       [6.4788578e+04, 2.8000000e+01, 6.4788578e+04],\n",
       "       [6.4567172e+04, 3.2000000e+01, 6.4567172e+04],\n",
       "       [6.6568508e+04, 0.0000000e+00, 6.6568508e+04],\n",
       "       [5.9192379e+04, 2.6000000e+01, 5.9192379e+04],\n",
       "       [6.3780352e+04, 2.0000000e+01, 6.3780352e+04],\n",
       "       [6.8936281e+04, 6.0000000e+00, 6.8936281e+04],\n",
       "       [6.1041871e+04, 4.0000000e+00, 6.1041871e+04],\n",
       "       [6.6730117e+04, 4.6000000e+01, 6.6730117e+04],\n",
       "       [6.2396461e+04, 4.8000000e+01, 6.2396461e+04],\n",
       "       [6.2696070e+04, 1.2000000e+01, 6.2696070e+04],\n",
       "       [6.5410629e+04, 2.8000000e+01, 6.5410629e+04],\n",
       "       [6.6272992e+04, 0.0000000e+00, 6.6272992e+04],\n",
       "       [6.7254469e+04, 3.2000000e+01, 6.7254469e+04],\n",
       "       [6.8090039e+04, 1.8000000e+01, 6.8090039e+04],\n",
       "       [6.9822391e+04, 3.4000000e+01, 6.9822391e+04],\n",
       "       [6.2538238e+04, 1.8000000e+01, 6.2538238e+04],\n",
       "       [6.1941191e+04, 8.0000000e+00, 6.1941191e+04],\n",
       "       [6.4560809e+04, 4.6000000e+01, 6.4560809e+04],\n",
       "       [6.4725109e+04, 1.0000000e+01, 6.4725109e+04],\n",
       "       [5.9343410e+04, 2.4000000e+01, 5.9343410e+04],\n",
       "       [6.7121680e+04, 2.0000000e+00, 6.7121680e+04],\n",
       "       [6.6863789e+04, 4.0000000e+01, 6.6863789e+04],\n",
       "       [6.0804031e+04, 1.8000000e+01, 6.0804031e+04],\n",
       "       [6.8438344e+04, 2.0000000e+00, 6.8438344e+04],\n",
       "       [6.8668078e+04, 2.8000000e+01, 6.8668078e+04],\n",
       "       [6.3116102e+04, 4.0000000e+01, 6.3116102e+04],\n",
       "       [6.9077609e+04, 8.0000000e+00, 6.9077609e+04],\n",
       "       [5.9373441e+04, 4.2000000e+01, 5.9373441e+04],\n",
       "       [6.3408551e+04, 4.8000000e+01, 6.3408551e+04],\n",
       "       [6.0769238e+04, 0.0000000e+00, 6.0769238e+04],\n",
       "       [5.9156551e+04, 3.0000000e+01, 5.9156551e+04],\n",
       "       [6.3727469e+04, 0.0000000e+00, 6.3727469e+04],\n",
       "       [6.8513430e+04, 2.4000000e+01, 6.8513430e+04],\n",
       "       [6.8335734e+04, 3.6000000e+01, 6.8335734e+04],\n",
       "       [6.8042742e+04, 3.8000000e+01, 6.8042742e+04],\n",
       "       [6.5921953e+04, 8.0000000e+00, 6.5921953e+04],\n",
       "       [6.6897750e+04, 4.4000000e+01, 6.6897750e+04],\n",
       "       [6.0400898e+04, 4.0000000e+01, 6.0400898e+04],\n",
       "       [6.9938117e+04, 2.6000000e+01, 6.9938117e+04],\n",
       "       [6.4383500e+04, 1.4000000e+01, 6.4383500e+04],\n",
       "       [7.0010953e+04, 4.4000000e+01, 7.0010953e+04],\n",
       "       [6.6803672e+04, 6.0000000e+00, 6.6803672e+04],\n",
       "       [6.7103328e+04, 4.4000000e+01, 6.7103328e+04],\n",
       "       [5.9685770e+04, 2.4000000e+01, 5.9685770e+04],\n",
       "       [6.0905000e+04, 8.0000000e+00, 6.0905000e+04],\n",
       "       [6.4716250e+04, 2.4000000e+01, 6.4716250e+04],\n",
       "       [6.1723609e+04, 4.2000000e+01, 6.1723609e+04],\n",
       "       [6.1547680e+04, 0.0000000e+00, 6.1547680e+04],\n",
       "       [6.8866648e+04, 4.2000000e+01, 6.8866648e+04],\n",
       "       [6.0094148e+04, 3.2000000e+01, 6.0094148e+04]], dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add an array with an array with reversed columns\n",
    "arr + arr[:,::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9daf58e0-b354-4dcc-9e5e-9ffe083e05b9",
   "metadata": {},
   "source": [
    "Putting everything together, we can do both together: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a5d4fde6-7147-42cc-9131-3755a353bf87",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 36676.812 , -23310.188 , -13366.628 ],\n",
       "       [ 39748.844 , -20238.154 , -19510.693 ],\n",
       "       [ 39154.36  , -20832.64  , -18321.72  ],\n",
       "       [ 37340.93  , -22653.072 , -14687.852 ],\n",
       "       [ 39164.42  , -20828.58  , -18335.84  ],\n",
       "       [ 36979.53  , -23004.467 , -13975.066 ],\n",
       "       [ 37994.945 , -22000.057 , -15994.887 ],\n",
       "       [ 38130.85  , -21875.146 , -16255.707 ],\n",
       "       [ 36820.33  , -23179.67  , -13640.66  ],\n",
       "       [ 38833.695 , -21155.303 , -17678.393 ],\n",
       "       [ 38863.84  , -21139.16  , -17724.68  ],\n",
       "       [ 39776.47  , -20212.533 , -19563.934 ],\n",
       "       [ 40170.562 , -19842.436 , -20328.125 ],\n",
       "       [ 37359.953 , -22638.049 , -14721.898 ],\n",
       "       [ 38466.203 , -21539.797 , -16926.406 ],\n",
       "       [ 38516.344 , -21497.656 , -17018.686 ],\n",
       "       [ 38793.273 , -21222.725 , -17570.555 ],\n",
       "       [ 38092.055 , -21925.943 , -16166.113 ],\n",
       "       [ 38864.08  , -21153.924 , -17710.154 ],\n",
       "       [ 39612.566 , -20392.434 , -19220.133 ],\n",
       "       [ 36705.83  , -23310.17  , -13395.66  ],\n",
       "       [ 40267.914 , -19754.088 , -20513.828 ],\n",
       "       [ 40012.656 , -19989.344 , -20023.314 ],\n",
       "       [ 40017.547 , -20004.453 , -20013.094 ],\n",
       "       [ 37577.28  , -22425.717 , -15151.566 ],\n",
       "       [ 37507.625 , -22497.377 , -15010.247 ],\n",
       "       [ 37971.594 , -22031.406 , -15940.186 ],\n",
       "       [ 38851.67  , -21170.33  , -17681.34  ],\n",
       "       [ 36920.953 , -23094.049 , -13826.898 ],\n",
       "       [ 37880.5   , -22128.5   , -15752.    ],\n",
       "       [ 37762.75  , -22260.248 , -15502.508 ],\n",
       "       [ 39050.383 , -20965.62  , -18084.76  ],\n",
       "       [ 40178.53  , -19830.47  , -20348.06  ],\n",
       "       [ 37607.773 , -22412.229 , -15195.539 ],\n",
       "       [ 37622.5   , -22399.5   , -15223.    ],\n",
       "       [ 37387.367 , -22636.633 , -14750.732 ],\n",
       "       [ 39679.195 , -20353.807 , -19325.387 ],\n",
       "       [ 39099.273 , -20931.727 , -18167.547 ],\n",
       "       [ 40277.805 , -19751.193 , -20526.613 ],\n",
       "       [ 38876.406 , -21159.594 , -17716.814 ],\n",
       "       [ 39376.727 , -20656.273 , -18720.453 ],\n",
       "       [ 37128.344 , -22896.654 , -14231.694 ],\n",
       "       [ 39782.992 , -20260.01  , -19522.98  ],\n",
       "       [ 40298.97  , -19741.03  , -20557.94  ],\n",
       "       [ 37233.633 , -22810.37  , -14423.259 ],\n",
       "       [ 39683.14  , -20342.857 , -19340.287 ],\n",
       "       [ 39275.746 , -20769.254 , -18506.494 ],\n",
       "       [ 39429.957 , -20610.043 , -18819.912 ],\n",
       "       [ 39320.055 , -20719.943 , -18600.113 ],\n",
       "       [ 38300.67  , -21733.33  , -16567.34  ],\n",
       "       [ 38450.14  , -21586.86  , -16863.28  ],\n",
       "       [ 38524.273 , -21511.725 , -17012.555 ],\n",
       "       [ 37863.5   , -22189.502 , -15673.992 ],\n",
       "       [ 40318.875 , -19722.127 , -20596.746 ],\n",
       "       [ 38791.547 , -21253.451 , -17538.102 ],\n",
       "       [ 37076.242 , -22976.76  , -14099.4795],\n",
       "       [ 39709.043 , -20345.957 , -19363.088 ],\n",
       "       [ 37806.96  , -22228.04  , -15578.919 ],\n",
       "       [ 39252.18  , -20782.82  , -18469.36  ],\n",
       "       [ 39159.312 , -20894.69  , -18264.62  ],\n",
       "       [ 38252.79  , -21794.209 , -16458.578 ],\n",
       "       [ 37971.    , -22090.998 , -15880.008 ],\n",
       "       [ 37639.508 , -22407.49  , -15232.02  ],\n",
       "       [ 37364.32  , -22690.68  , -14673.64  ],\n",
       "       [ 36785.203 , -23262.797 , -13522.407 ],\n",
       "       [ 39216.92  , -20840.08  , -18376.84  ],\n",
       "       [ 39418.6   , -20644.396 , -18774.207 ],\n",
       "       [ 38540.062 , -21504.936 , -17035.125 ],\n",
       "       [ 38492.297 , -21571.703 , -16920.594 ],\n",
       "       [ 40284.863 , -19773.137 , -20511.727 ],\n",
       "       [ 37696.773 , -22373.227 , -15323.547 ],\n",
       "       [ 37777.406 , -22274.596 , -15502.806 ],\n",
       "       [ 39801.992 , -20262.01  , -19539.98  ],\n",
       "       [ 37260.883 , -22812.115 , -14448.775 ],\n",
       "       [ 37180.977 , -22880.025 , -14300.945 ],\n",
       "       [ 39030.633 , -21025.367 , -18005.268 ],\n",
       "       [ 37049.797 , -23023.203 , -14026.593 ],\n",
       "       [ 40279.85  , -19777.146 , -20502.707 ],\n",
       "       [ 38934.816 , -21120.184 , -17814.633 ],\n",
       "       [ 39823.586 , -20256.412 , -19567.172 ],\n",
       "       [ 40357.15  , -19708.85  , -20648.299 ],\n",
       "       [ 38839.508 , -21242.49  , -17597.02  ],\n",
       "       [ 37241.188 , -22829.81  , -14411.381 ],\n",
       "       [ 37299.42  , -22766.578 , -14532.848 ],\n",
       "       [ 37397.75  , -22668.248 , -14729.508 ],\n",
       "       [ 38110.68  , -21971.318 , -16139.368 ],\n",
       "       [ 37780.414 , -22284.584 , -15495.834 ],\n",
       "       [ 39947.703 , -20120.299 , -19827.398 ],\n",
       "       [ 36771.96  , -23304.04  , -13467.919 ],\n",
       "       [ 38626.5   , -21456.5   , -17170.    ],\n",
       "       [ 36746.68  , -23322.318 , -13424.368 ],\n",
       "       [ 37823.11  , -22265.89  , -15557.221 ],\n",
       "       [ 37717.89  , -22353.11  , -15364.779 ],\n",
       "       [ 40194.742 , -19887.256 , -20307.486 ],\n",
       "       [ 39792.    , -20299.    , -19493.    ],\n",
       "       [ 38519.914 , -21564.084 , -16955.834 ],\n",
       "       [ 39515.46  , -20560.537 , -18954.928 ],\n",
       "       [ 39582.11  , -20515.893 , -19066.213 ],\n",
       "       [ 37136.453 , -22941.549 , -14194.898 ],\n",
       "       [ 40063.28  , -20020.717 , -20042.566 ]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# subtract each element from the average of its row\n",
    "arr - np.mean(arr, axis=-1, keepdims=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b796fd3-19f2-4bf4-b872-7d58c65dc0c7",
   "metadata": {},
   "source": [
    "Some additional NumPy operations Ponder currently supports include:\n",
    "\n",
    "- Element-wise matrix operations such as addition, subtraction, multiplication, division, power\n",
    "- Axis-collapsing or reducing operations such as min, max, sum, product, mean\n",
    "- Multi-array operations such as maximum or minimum\n",
    "- And many others, such as where, ravel, and transpose"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a58431d9-60fe-4cfd-84a3-5ef8bc1f29ec",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this tutorial, we demonstrated how you can Ponder to run NumPy operations natively directly on your database. Congrats! This wraps up our 8-part tutorial series on how you can use Ponder to start accelerating your data science workflow. We're so excited to see what you can do with Ponder! If you have any questions, issues, or want to show off something cool you've built with Ponder:\n",
    "- [Join us on Slack](https://modin.org/slack.html) and share your thoughts on #ponder-support, or\n",
    "- Drop us a note at support@ponder.io."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7f3c82ee-0b42-46e6-8fc8-34cf15e8958b",
   "metadata": {},
   "outputs": [],
   "source": [
    "duckdb_con.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}