CTE deprecation

CHANGELOG.md

@@ -3,6 +3,8 @@
 ## 0.7.8dev
 
 * [Feature] Add `%sqlplot bar` and `%sqlplot pie`
+* [Feature] Automated dependency inference when creating CTEs. `--with` is now deprecated and will display a warning. (#166)
+
 
 ## 0.7.7 (2023-05-31)
 

doc/_toc.yml

@@ -42,6 +42,7 @@ parts:
     - file: api/magic-sql
     - file: api/magic-plot
     - file: api/magic-render
+    - file: api/magic-snippets
     - file: api/configuration
     - file: api/python
     - file: api/magic-tables-columns

doc/api/magic-snippets.md

@@ -0,0 +1,135 @@
+---
+jupytext:
+  notebook_metadata_filter: myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.14.5
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+myst:
+  html_meta:
+    description lang=en: Documentation for  %sqlcmd snippets
+      from JupySQL
+    keywords: jupyter, sql, jupysql, snippets
+    property=og:locale: en_US
+---
+
+# `%sqlcmd snippets`
+
+`%sqlcmd snippets` returns the query snippets saved using `--save`
+
+## Load Data
+
+```{code-cell} ipython3
+%load_ext sql
+%sql duckdb://
+```
+
+```{code-cell} ipython3
+from pathlib import Path
+from urllib.request import urlretrieve
+
+if not Path("penguins.csv").is_file():
+    urlretrieve(
+        "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv",
+        "penguins.csv",
+    )
+```
+
+```{code-cell} ipython3
+%%sql
+SELECT * FROM penguins.csv LIMIT 3
+```
+
+Let's save a couple of snippets.
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%%sql --save gentoo
+SELECT * FROM penguins.csv where species == 'Gentoo'
+```
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%%sql --save chinstrap
+SELECT * FROM penguins.csv where species == 'Chinstrap'
+```
+
+## `%sqlcmd snippets`
+
++++
+
+Returns all the snippets saved in the environment
+
+```{code-cell} ipython3
+%sqlcmd snippets
+```
+
+Arguments:
+
+`-d`/`--delete` Delete a snippet.
+
+`-D`/`--delete-force` Force delete a snippet. This may be useful if there are other dependent snippets, and you still need to delete this snippet.
+
+`-A`/`--delete-force-all` Force delete a snippet and all dependent snippets.
+
+```{code-cell} ipython3
+
+%sqlcmd snippets -d gentoo
+```
+
+This deletes the stored snippet `gentoo`.
+
+To demonstrate `force-delete` let's create a snippet dependent on `chinstrap` snippet.
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%%sql --save chinstrap_sub
+SELECT * FROM chinstrap where island == 'Dream'
+```
++++
+
+Trying to delete the `chinstrap` snippet will display an error message:
+
+```{code-cell} ipython3
+:tags: [raises-exception]
+
+%sqlcmd snippets -d chinstrap
+```
+
+If you still wish to delete this snippet, you can run the below command:
+
+```{code-cell} ipython3
+
+%sqlcmd snippets -D chinstrap
+```
+
+Now, let's see how to delete a snippet and all other dependent snippets. We'll create a few snippets again.
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%%sql --save chinstrap
+SELECT * FROM penguins.csv where species == 'Chinstrap'
+```
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%%sql --save chinstrap_sub
+SELECT * FROM chinstrap where island == 'Dream'
+```
+
+Now, force delete `chinstrap` and its dependent `chinstrap_sub`:
+
+```{code-cell} ipython3
+
+%sqlcmd snippets -A chinstrap
+```

doc/compose.md

@@ -27,7 +27,8 @@ pip install jupysql matplotlib
 ```
 
 
-*New in version 0.4.3*
+```{versionchanged} 0.7.8
+```
 
 ```{note}
 This is a beta feature, please [join our community](https://ploomber.io/community) and
@@ -105,31 +106,39 @@ OR Name LIKE '%metal%'
 
 Join the filtered genres and tracks, so we only get Rock and Metal tracks, and save the query as `track_fav`
 
-Note that we are using `--with`; this will retrieve previously saved queries, and prepend them (using CTEs), then, we save the query in `track_fav` .
+
+We automatically extract the tables from the query and infer the dependencies from all the saved snippets.
+
 
 ```{code-cell} ipython3
-%%sql --with genres_fav --with tracks_with_info --save track_fav
+%%sql --save track_fav
 SELECT t.*
 FROM tracks_with_info t
 JOIN genres_fav
 ON t.GenreId = genres_fav.GenreId
 ```
 
-Use the `track_fav` query to find artists with the most Rock and Metal tracks, and save the query as `top_artist`
+Now let's find artists with the most Rock and Metal tracks, and save the query as `top_artist`
 
 ```{code-cell} ipython3
-%%sql --with track_fav --save top_artist
+%%sql --save top_artist
 SELECT artist, COUNT(*) FROM track_fav
 GROUP BY artist
 ORDER BY COUNT(*) DESC
 ```
 
+
+```{note}
+A saved snippet will override an existing table with the same name during query formation. If you wish to delete a snippet please refer to [sqlcmd snippets API](api/magic-snippets.md).
+
+```
+
 #### Data visualization
 
 Once we have the desired results from the query `top_artist`, we can generate a visualization using the bar method
 
 ```{code-cell} ipython3
-top_artist = %sql --with top_artist SELECT * FROM top_artist
+top_artist = %sql SELECT * FROM top_artist
 top_artist.bar()
 ```
 

doc/plot.md

@@ -121,10 +121,10 @@ FROM "yellow_tripdata_2021-01.parquet"
 WHERE trip_distance < 6.3
 ```
 
-Now, let's plot again, but this time let's pass `--table short_trips`. Note that this table *doesn't exist*; however, since we're passing the `--with` argument, JupySQL will use the query we defined above:
+Now, let's plot again, but this time let's pass `--table short_trips`. Note that this table *doesn't exist*; JupySQL will automatically infer and use the saved snippet defined above.
 
 ```{code-cell} ipython3
-%sqlplot boxplot --table short_trips --column trip_distance --with short_trips
+%sqlplot boxplot --table short_trips --column trip_distance
 ```
 
 We can see the highest value is a bit over 6, that's expected since we set a 6.3 cutoff value.
@@ -133,18 +133,18 @@ We can see the highest value is a bit over 6, that's expected since we set a 6.3
 
 ## Histogram
 
-To create a histogram, call `%sqlplot histogram`, and pass the name of the table, the column you want to plot, and the number of bins. Similarly to what we did in the [Boxplot](#boxplot) example, we're using `--with short_trips` so JupySQL uses the query we defined and only plots such data subset.
+To create a histogram, call `%sqlplot histogram`, and pass the name of the table, the column you want to plot, and the number of bins. Similarly to what we did in the [Boxplot](#boxplot) example, JupySQL detects a saved snippet and only plots such data subset.
 
 ```{code-cell} ipython3
-%sqlplot histogram --table short_trips --column trip_distance --bins 10 --with short_trips
+%sqlplot histogram --table short_trips --column trip_distance --bins 10
 ```
 
 ## Customize plot
 
 `%sqlplot` returns a `matplotlib.Axes` object that you can further customize:
 
 ```{code-cell} ipython3
-ax = %sqlplot histogram --table short_trips --column trip_distance --bins 50 --with short_trips
+ax = %sqlplot histogram --table short_trips --column trip_distance --bins 50
 ax.grid()
 ax.set_title("Trip distance from trips < 6.3")
 _ = ax.set_xlabel("Trip distance")

src/sql/command.py

@@ -71,10 +71,6 @@ def __init__(self, magic, user_ns, line, cell) -> None:
         if add_alias:
             self.parsed["connection"] = self.args.line[0]
 
-        if self.args.with_:
-            final = store.render(self.parsed["sql"], with_=self.args.with_)
-            self.parsed["sql"] = str(final)
-
     @property
     def sql(self):
         """
@@ -112,3 +108,15 @@ def __repr__(self) -> str:
             f"{type(self).__name__}(line={self._line!r}, cell={self._cell!r}) -> "
             f"({self.sql!r}, {self.sql_original!r})"
         )
+
+    def set_sql_with(self, with_):
+        """
+        Sets the final rendered SQL query using the WITH clause
+
+        Parameters
+        ----------
+        with_ : list
+        list of all subqueries needed to render the query
+        """
+        final = store.render(self.parsed["sql"], with_)
+        self.parsed["sql"] = str(final)

src/sql/error_message.py

@@ -5,6 +5,33 @@
 ORIGINAL_ERROR = "\nOriginal error message from DB driver:\n"
 
 
+def parse_sqlglot_error(e, q):
+    """
+    Function to parse the error message from sqlglot
+
+    Parameters
+    ----------
+    e: sqlglot.errors.ParseError, exception
+            while parsing through sqlglot
+    q : str, user query
+
+    Returns
+    -------
+    str
+        Formatted error message containing description
+        and positions
+    """
+    err = e.errors
+    position = ""
+    for item in err:
+        position += (
+            f"Syntax Error in {q}: {item['description']} at "
+            f"Line {item['line']}, Column {item['col']}\n"
+        )
+    msg = "Possible reason: \n" + position if position else ""
+    return msg
+
+
 def detail(original_error, query=None):
     original_error = str(original_error)
     return_msg = SYNTAX_ERROR
@@ -25,18 +52,8 @@ def detail(original_error, query=None):
                 )
 
             except sqlglot.errors.ParseError as e:
-                err = e.errors
-                position = ""
-                for item in err:
-                    position += (
-                        f"Syntax Error in {q}: {item['description']} at "
-                        f"Line {item['line']}, Column {item['col']}\n"
-                    )
-                return_msg = (
-                    return_msg + "Possible reason: \n" + position
-                    if position
-                    else return_msg
-                )
+                parse_msg = parse_sqlglot_error(e, q)
+                return_msg = return_msg + parse_msg if parse_msg else return_msg
 
         return return_msg + "\n" + ORIGINAL_ERROR + original_error + "\n"