Adding support for SQL as execution engine

.github/workflows/python-app.yml

@@ -15,6 +15,22 @@ jobs:
 
     runs-on: ubuntu-latest
 
+    # Service containers to run with `container-job`
+    services:
+      # Label used to access the service container
+      postgres:
+        # Docker Hub image
+        image: postgres
+        # Provide the password for postgres
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: lux
+          POSTGRES_DB: postgres
+        # Set health checks to wait until postgres has started
+        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
+        ports:
+          - 5432:5432
+
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python 3.7
@@ -28,6 +44,11 @@ jobs:
         pip install wheel
         pip install -r requirements.txt
         pip install -r requirements-dev.txt
+        pip install sqlalchemy
+    - name: Upload data to Postgres
+      run: |
+        python lux/data/upload_car_data.py
+        python lux/data/upload_aug_test_data.py
     - name: Lint check with black
       run: |
         black --target-version py37 --line-length 105 --check .

CONTRIBUTING.md

@@ -37,7 +37,7 @@ lux/
 ```
 
 # Code Formatting
-In order to keep our codebase clean and readible, we are using PEP8 guidelines. To help us maintain and check code style, we are using [black](https://github.com/psf/black). Simply run `black .` before commiting. Failure to do so may fail the tests run on Travis. This package should have been installed for you as part of [requirements-dev](https://github.com/lux-org/lux/blob/master/requirements-dev.txt). 
+In order to keep our codebase clean and readible, we are using PEP8 guidelines. To help us maintain and check code style, we are using [black](https://github.com/psf/black). Simply run `black .` before commiting. Without running black,  the checks on the continuous integration tests can fail. `black` should be installed for you as part of [requirements-dev](https://github.com/lux-org/lux/blob/master/requirements-dev.txt). 
 
 # Running the Test Suite
 
@@ -55,8 +55,6 @@ To run a single test file, run:
 python -m pytest tests/<test_file_name>.py
 ```
 
-
-
 # Submitting a Pull Request
 
  You can commit your code and push to your forked repo. Once all of your local changes have been tested and formatted, you are ready to submit a PR. For Lux, we use the "Squash and Merge" strategy to merge in PR, which means that even if you make a lot of small commits in your PR, they will all get squashed into a single commit associated with the PR. Please make sure that comments and unnecessary file changes are not committed as part of the PR by looking at the "File Changes" diff view on the pull request page.

doc/source/advanced/executor.rst

@@ -11,41 +11,57 @@ Please refer to :mod:`lux.executor.Executor`, if you are interested in extending
 SQL Executor
 =============
 
-Lux extends its visualization exploration operations to data within SQL databases. By using the SQL Executor, users can specify a SQL database to connect a Lux Dataframe for generating all the visualizations recommended in Lux.
+Lux extends its visualization exploration operations to data within SQL databases. By using the SQL Executor, users can specify a SQL database to connect a LuxSQLTable for generating all the visualizations recommended in Lux.
 
 Connecting Lux to a Database
 ----------------------------
 
-Before Lux can operate on data within a Postgresql database, users have to connect their Lux Dataframe to their database.
+Before Lux can operate on data within a Postgresql database, users have to connect their LuxSQLTable to their database.
 To do this, users first need to specify a connection to their SQL database. This can be done using the psycopg2 package's functionality.
 
 .. code-block:: python
 
 	import psycopg2
 	connection = psycopg2.connect("dbname=example_database user=example_user, password=example_password")
 
-Once this connection is created, users can connect their Lux Dataframe to the database using the Lux Dataframe's set_SQL_connection command.
+Once this connection is created, users can connect the lux config to the database using the set_SQL_connection command.
 
 .. code-block:: python
 
-	lux_df.set_SQL_connection(connection, "my_table")
+	lux.config.set_SQL_connection(connection)
 
-When the set_SQL_connection function is called, Lux will then populate the Dataframe with all the metadata it needs to run its intent from the database table. 
+When the set_SQL_connection function is called, Lux will then populate the LuxSQLTable with all the metadata it needs to run its intent from the database table. 
+
+Connecting a LuxSQLTable to a Table/View
+--------------------------
+
+LuxSQLTables can be connected to individual tables or views created within your Postgresql database. This can be done by either specifying the table/view name in the constructor.
+
+.. code-block:: python
+
+	sql_tbl = LuxSQLTable(table_name = "my_table")
+
+You can also connect a LuxSQLTable to a table/view by using the set_SQL_table function.
+
+.. code-block:: python
+
+	sql_tbl = LuxSQLTable()
+	sql_tbl.set_SQL_table("my_table")
 
 Choosing an Executor
 --------------------------
 
 Once a user has created a connection to their Postgresql database, they need to change Lux's execution engine so that the system can collect and process the data properly.
-By default Lux uses the Pandas executor to process local data in the Lux Dataframe, but users need to use the SQL executor when their Lux Dataframe is connected to a database.
-Users can specify the executor that a Lux Dataframe will use via the set_executor_type function as follows:
+By default Lux uses the Pandas executor to process local data in the LuxDataframe, but users will use the SQL executor when their LuxSQLTable is connected to a database.
+Users can specify the executor that Lux will use via the set_executor_type function as follows:
 
 .. code-block:: python
 
 	lux_df.set_executor_type("SQL")
 
-Once a Lux Dataframe has been connected to a Postgresql table and set to use the SQL Executor, users can take full advantage of Lux's visual exploration capabilities as-is. Users can set their intent to specify which variables they are most interested in and discover insightful visualizations from their database.
+Once a LuxSQLTable has been connected to a Postgresql table and set to use the SQL Executor, users can take full advantage of Lux's visual exploration capabilities as-is. Users can set their intent to specify which variables they are most interested in and discover insightful visualizations from their database.
 
 SQL Executor Limitations
 --------------------------
 
-While users can make full use of Lux's functionalities on data within a database table, they will not be able to use any of Pandas' Dataframe functions to manipulate the data. Since the Lux SQL Executor delegates most data processing to the Postgresql database, it does not pull in the entire dataset into the Lux Dataframe. As such there is no actual data within the Lux Dataframe to manipulate, only the relevant metadata required to for Lux to manage its intent. Thus, if users are interested in manipulating or querying their data, this needs to be done through SQL or an alternative RDBMS interface.
+While users can make full use of Lux's functionalities on data within a database table, they will not be able to use any of Pandas' Dataframe functions to manipulate the data in the LuxSQLTable object. Since the Lux SQL Executor delegates most data processing to the Postgresql database, it does not pull in the entire dataset into the Lux Dataframe. As such there is no actual data within the LuxSQLTable to manipulate, only the relevant metadata required to for Lux to manage its intent. Thus, if users are interested in manipulating or querying their data, this needs to be done through SQL or an alternative RDBMS interface.

lux/__init__.py

@@ -15,6 +15,7 @@
 # Register the commonly used modules (similar to how pandas does it: https://github.com/pandas-dev/pandas/blob/master/pandas/__init__.py)
 from lux.vis.Clause import Clause
 from lux.core.frame import LuxDataFrame
+from lux.core.sqltable import LuxSQLTable
 from ._version import __version__, version_info
 from lux._config import config
 from lux._config.config import warning_format

lux/_config/config.py

@@ -343,6 +343,7 @@ def set_SQL_connection(self, connection):
             connection : SQLAlchemy connectable, str, or sqlite3 connection
                 For more information, `see here <https://docs.sqlalchemy.org/en/13/core/connections.html>`__
         """
+        self.set_executor_type("SQL")
         self.SQLconnection = connection
 
     def set_executor_type(self, exe):
@@ -358,10 +359,13 @@ def set_executor_type(self, exe):
             from lux.executor.SQLExecutor import SQLExecutor
 
             self.executor = SQLExecutor()
-        else:
+        elif exe == "Pandas":
             from lux.executor.PandasExecutor import PandasExecutor
 
+            self.SQLconnection = ""
             self.executor = PandasExecutor()
+        else:
+            raise ValueError("Executor type must be either 'Pandas' or 'SQL'")
 
 
 def warning_format(message, category, filename, lineno, file=None, line=None):

lux/action/correlation.py

@@ -73,7 +73,6 @@ def correlation(ldf: LuxDataFrame, ignore_transpose: bool = True):
             )
         msr1 = measures[0].attribute
         msr2 = measures[1].attribute
-
         if ignore_transpose:
             check_transpose = check_transpose_not_computed(vlist, msr1, msr2)
         else:

lux/action/custom.py

@@ -64,7 +64,7 @@ def custom_actions(ldf):
     recommendations : Dict[str,obj]
         object with a collection of visualizations that were previously registered.
     """
-    if len(lux.config.actions) > 0 and len(ldf) > 0:
+    if len(lux.config.actions) > 0 and (len(ldf) > 0 or lux.config.executor.name != "PandasExecutor"):
         recommendations = []
         for action_name in lux.config.actions.keys():
             display_condition = lux.config.actions[action_name].display_condition

lux/core/frame.py

@@ -18,6 +18,7 @@
 from lux.vis.Vis import Vis
 from lux.vis.VisList import VisList
 from lux.history.history import History
+from lux.utils.date_utils import is_datetime_series
 from lux.utils.message import Message
 from lux.utils.utils import check_import_lux_widget
 from typing import Dict, Union, List, Callable
@@ -57,8 +58,6 @@ class LuxDataFrame(pd.DataFrame):
     ]
 
     def __init__(self, *args, **kw):
-        from lux.executor.PandasExecutor import PandasExecutor
-
         self._history = History()
         self._intent = []
         self._inferred_intent = []
@@ -70,7 +69,14 @@ def __init__(self, *args, **kw):
         super(LuxDataFrame, self).__init__(*args, **kw)
 
         self.table_name = ""
-        lux.config.executor = PandasExecutor()
+        if lux.config.SQLconnection == "":
+            from lux.executor.PandasExecutor import PandasExecutor
+
+            lux.config.executor = PandasExecutor()
+        else:
+            from lux.executor.SQLExecutor import SQLExecutor
+
+            lux.config.executor = SQLExecutor()
 
         self._sampled = None
         self._toggle_pandas_display = True
@@ -110,14 +116,25 @@ def data_type(self):
         return self._data_type
 
     def maintain_metadata(self):
+        is_sql_tbl = lux.config.executor.name == "SQLExecutor"
+        if lux.config.SQLconnection != "" and is_sql_tbl:
+            from lux.executor.SQLExecutor import SQLExecutor
+
+            lux.config.executor = SQLExecutor()
+
         # Check that metadata has not yet been computed
         if not hasattr(self, "_metadata_fresh") or not self._metadata_fresh:
             # only compute metadata information if the dataframe is non-empty
-            if len(self) > 0:
-                lux.config.executor.compute_stats(self)
+            if is_sql_tbl:
                 lux.config.executor.compute_dataset_metadata(self)
                 self._infer_structure()
                 self._metadata_fresh = True
+            else:
+                if len(self) > 0:
+                    lux.config.executor.compute_stats(self)
+                    lux.config.executor.compute_dataset_metadata(self)
+                    self._infer_structure()
+                    self._metadata_fresh = True
 
     def expire_recs(self):
         """
@@ -168,12 +185,14 @@ def _infer_structure(self):
         # If the dataframe is very small and the index column is not a range index, then it is likely that this is an aggregated data
         is_multi_index_flag = self.index.nlevels != 1
         not_int_index_flag = not pd.api.types.is_integer_dtype(self.index)
-        small_df_flag = len(self) < 100
+        is_sql_tbl = lux.config.executor.name == "SQLExecutor"
+
+        small_df_flag = len(self) < 100 and is_sql_tbl
         if self.pre_aggregated == None:
             self.pre_aggregated = (is_multi_index_flag or not_int_index_flag) and small_df_flag
             if "Number of Records" in self.columns:
                 self.pre_aggregated = True
-            self.pre_aggregated = "groupby" in [event.name for event in self.history]
+            self.pre_aggregated = "groupby" in [event.name for event in self.history] and not is_sql_tbl
 
     @property
     def intent(self):
@@ -317,110 +336,6 @@ def current_vis(self):
     def current_vis(self, current_vis: Dict):
         self._current_vis = current_vis
 
-    #######################################################
-    ########## SQL Metadata, type, model schema ###########
-    #######################################################
-
-    def set_SQL_table(self, t_name):
-        self.table_name = t_name
-        self.compute_SQL_dataset_metadata()
-
-    def compute_SQL_dataset_metadata(self):
-        self.get_SQL_attributes()
-        for attr in list(self.columns):
-            self[attr] = None
-        self._data_type = {}
-        #####NOTE: since we aren't expecting users to do much data processing with the SQL database, should we just keep this
-        #####      in the initialization and do it just once
-        self.compute_SQL_data_type()
-        self.compute_SQL_stats()
-
-    def compute_SQL_stats(self):
-        # precompute statistics
-        self.unique_values = {}
-        self._min_max = {}
-
-        self.get_SQL_unique_values()
-        # self.get_SQL_cardinality()
-        for attribute in self.columns:
-            if self._data_type[attribute] == "quantitative":
-                self._min_max[attribute] = (
-                    self[attribute].min(),
-                    self[attribute].max(),
-                )
-
-    def get_SQL_attributes(self):
-        if "." in self.table_name:
-            table_name = self.table_name[self.table_name.index(".") + 1 :]
-        else:
-            table_name = self.table_name
-        query = f"SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS where TABLE_NAME = '{table_name}'"
-        attributes = list(pd.read_sql(query, lux.config.SQLconnection)["column_name"])
-        for attr in attributes:
-            self[attr] = None
-
-    def get_SQL_cardinality(self):
-        cardinality = {}
-        for attr in list(self.columns):
-            card_query = pd.read_sql(
-                f"SELECT Count(Distinct({attr})) FROM {self.table_name}",
-                lux.config.SQLconnection,
-            )
-            cardinality[attr] = list(card_query["count"])[0]
-        self.cardinality = cardinality
-
-    def get_SQL_unique_values(self):
-        unique_vals = {}
-        for attr in list(self.columns):
-            unique_query = pd.read_sql(
-                f"SELECT Distinct({attr}) FROM {self.table_name}",
-                lux.config.SQLconnection,
-            )
-            unique_vals[attr] = list(unique_query[attr])
-        self.unique_values = unique_vals
-
-    def compute_SQL_data_type(self):
-        data_type = {}
-        sql_dtypes = {}
-        self.get_SQL_cardinality()
-        if "." in self.table_name:
-            table_name = self.table_name[self.table_name.index(".") + 1 :]
-        else:
-            table_name = self.table_name
-        # get the data types of the attributes in the SQL table
-        for attr in list(self.columns):
-            query = f"SELECT DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '{table_name}' AND COLUMN_NAME = '{attr}'"
-            datatype = list(pd.read_sql(query, lux.config.SQLconnection)["data_type"])[0]
-            sql_dtypes[attr] = datatype
-
-        for attr in list(self.columns):
-            if attr in self._type_override:
-                data_type[attr] = self._type_override[attr]
-            elif str(attr).lower() in ["month", "year"]:
-                data_type[attr] = "temporal"
-            elif sql_dtypes[attr] in [
-                "character",
-                "character varying",
-                "boolean",
-                "uuid",
-                "text",
-            ]:
-                data_type[attr] = "nominal"
-            elif sql_dtypes[attr] in [
-                "integer",
-                "real",
-                "smallint",
-                "smallserial",
-                "serial",
-            ]:
-                if self.cardinality[attr] < 13:
-                    data_type[attr] = "nominal"
-                else:
-                    data_type[attr] = "quantitative"
-            elif "time" in sql_dtypes[attr] or "date" in sql_dtypes[attr]:
-                data_type[attr] = "temporal"
-        self._data_type = data_type
-
     def _append_rec(self, rec_infolist, recommendations: Dict):
         if recommendations["collection"] is not None and len(recommendations["collection"]) > 0:
             rec_infolist.append(recommendations)
@@ -470,6 +385,7 @@ def maintain_recs(self, is_series="DataFrame"):
 
         # Check that recs has not yet been computed
         if not hasattr(rec_df, "_recs_fresh") or not rec_df._recs_fresh:
+            is_sql_tbl = lux.config.executor.name == "SQLExecutor"
             rec_infolist = []
             from lux.action.row_group import row_group
             from lux.action.column_group import column_group
@@ -479,7 +395,7 @@ def maintain_recs(self, is_series="DataFrame"):
                 if rec_df.columns.name is not None:
                     rec_df._append_rec(rec_infolist, row_group(rec_df))
                 rec_df._append_rec(rec_infolist, column_group(rec_df))
-            elif not (len(rec_df) < 5 and not rec_df.pre_aggregated) and not (
+            elif not (len(rec_df) < 5 and not rec_df.pre_aggregated and not is_sql_tbl) and not (
                 self.index.nlevels >= 2 or self.columns.nlevels >= 2
             ):
                 from lux.action.custom import custom_actions

lux/core/series.py

@@ -43,13 +43,13 @@ class LuxSeries(pd.Series):
         "_prev",
         "_history",
         "_saved_export",
+        "name",
         "_sampled",
         "_toggle_pandas_display",
         "_message",
         "_pandas_only",
         "pre_aggregated",
         "_type_override",
-        "name",
     ]
 
     _default_metadata = {