parent aaf4021

author Eduardo Blancas <github@blancas.io> 1706144090 -0600
committer neelasha23 <neelasha.sen@gmail.com> 1706161587 +0530

parent aaf4021
author Eduardo Blancas <github@blancas.io> 1706144090 -0600
committer neelasha23 <neelasha.sen@gmail.com> 1706161586 +0530

parent aaf4021
author Eduardo Blancas <github@blancas.io> 1706144090 -0600
committer neelasha23 <neelasha.sen@gmail.com> 1706161584 +0530

parent aaf4021
author Eduardo Blancas <github@blancas.io> 1706144090 -0600
committer neelasha23 <neelasha.sen@gmail.com> 1706161581 +0530

Update README.md

arg expansion guide

docstring modified

doc fix

minor fix

supported args

table

orient

Parse line

new util func

rendering string

fix test

Removed table

with

doc fix

magic_sql

removed comment

removed comment

docstring

Lint

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## 0.10.8dev
 * [Fix] Fix edge case where `select` and other SQL keywords were not properly used to find where the user's query started, causing argument parsing issues (#973)
+* [Feature] Add support for parametrizing string type arguments of `%%sql`, `%sqlplot`, `%sqlcmd`' (#699)
 
 ## 0.10.7 (2023-12-23)
 

doc/_toc.yml

@@ -13,6 +13,7 @@ parts:
     - file: user-guide/tables-columns
     - file: user-guide/ggplot
     - file: user-guide/template
+    - file: user-guide/argument-expansion
     - file: user-guide/connection-file
     - file: user-guide/table_explorer
     - file: user-guide/data-profiling

doc/api/magic-plot.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.15.1
+    jupytext_version: 1.16.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -293,3 +293,50 @@ You can also show the percentage on top of the pie using the `S`/`show-numbers`
 ```{code-cell} ipython3
 %sqlplot pie --table penguins.csv --column species -S
 ```
+
+## Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+```{code-cell} ipython3
+%%sql 
+DROP TABLE IF EXISTS penguins;
+CREATE SCHEMA IF NOT EXISTS s1;
+CREATE TABLE s1.penguins (
+    species VARCHAR(255),
+    island VARCHAR(255),
+    bill_length_mm DECIMAL(5, 2),
+    bill_depth_mm DECIMAL(5, 2),
+    flipper_length_mm DECIMAL(5, 2),
+    body_mass_g INTEGER,
+    sex VARCHAR(255)
+);
+COPY s1.penguins FROM 'penguins.csv' WITH (FORMAT CSV, HEADER TRUE);
+```
+
+```{code-cell} ipython3
+table = "penguins"
+schema = "s1"
+orient = "h"
+column = "bill_length_mm"
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{table}} --schema {{schema}} --column {{column}} --orient {{orient}}
+```
+
+Now let's see another example using `--with`:
+
+```{code-cell} ipython3
+snippet = "gentoo"
+```
+
+```{code-cell} ipython3
+%%sql --save {{snippet}}
+SELECT * FROM {{schema}}.{{table}} 
+WHERE species == 'Gentoo'
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{snippet}} --with {{snippet}} --column {{column}}
+```

doc/api/magic-profile.md

@@ -201,4 +201,28 @@ Let’s profile `my_numbers` of `b_schema`
 
 ```{code-cell} ipython3
 %sqlcmd profile --table my_numbers --schema b_schema
-```
+```
+
+# Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's look at an example that uses variable expansion for `table`, `schema` and `output` arguments:
+
+```{code-cell} ipython3
+table = "my_numbers"
+schema = "b_schema"
+output = "numbers-report.html"
+```
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%sqlcmd profile --table {{table}} --schema {{schema}} --output {{output}}
+```
+
+```{code-cell} ipython3
+from IPython.display import HTML
+
+HTML(output)
+```

doc/api/magic-sql.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.15.1
+    jupytext_version: 1.16.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -359,3 +359,21 @@ LIMIT 3
 ```{code-cell} ipython3
 %sql --file my-query.sql
 ```
+
+## Parameterizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's see an example of creating a connection using an alias and closing the same through variable substitution.
+
+```{code-cell} ipython3
+alias = "db-four"
+```
+
+```{code-cell} ipython3
+%sql sqlite:///db_four.db --alias {{alias}}
+```
+
+```{code-cell} ipython3
+%sql --close {{alias}}
+```

doc/user-guide/argument-expansion.md

@@ -0,0 +1,108 @@
+---
+jupytext:
+  notebook_metadata_filter: myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.14.7
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+myst:
+  html_meta:
+    description lang=en: Variable substitution of arguments in Jupyter via JupySQL
+    keywords: jupyter, sql, jupysql, jinja
+    property=og:locale: en_US
+---
+
+# Parameterizing arguments
+
+```{versionadded} 0.10.8
+JupySQL uses Jinja templates for enabling parametrization of arguments. Arguments are parametrized with `{{variable}}`.
+```
+
+
+## Parametrization via `{{variable}}`
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+The benefits of using parametrized arguments is that they can be reused for different purposes.
+
+Let's load some data and connect to the in-memory DuckDB instance:
+
+```{code-cell} ipython3
+%load_ext sql
+%sql duckdb://
+%config SqlMagic.displaylimit = 3
+```
+
+```{code-cell} ipython3
+filename = "penguins.csv"
+```
+
+
+```{code-cell} ipython3
+from pathlib import Path
+from urllib.request import urlretrieve
+
+if not Path("penguins.csv").is_file():
+    urlretrieve(
+        "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv",
+        filename,
+    )
+```
+
+Now let's create a snippet from the data by declaring a `table` variable and use it in the `--save` argument.
+
++++
+
+### Create a snippet
+
+```{code-cell} ipython3
+table = "penguins_data"
+```
+
+```{code-cell} ipython3
+%%sql --save {{table}}
+SELECT *
+FROM penguins.csv
+```
+
+```{code-cell} ipython3
+snippet = %sqlcmd snippets {{table}}
+print(snippet)
+```
+
+
+### Plot a histogram
+
+Now, let's declare a variable `column` and plot a histogram on the data.
+
+```{code-cell} ipython3
+column = "body_mass_g"
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{table}} --column {{column}}
+```
+
+### Profile and Explore
+
+We can use the `filename` variable to profile and explore the data as well:
+
+```{code-cell} ipython3
+%sqlcmd profile --table {{filename}}
+```
+
+```{code-cell} ipython3
+%sqlcmd explore --table {{filename}}
+```
+
+### Run some tests
+
+```{code-cell} ipython3
+%sqlcmd test --table {{table}} --column {{column}} --greater 3500
+```
+

src/sql/cmd/columns.py

@@ -1,20 +1,25 @@
 from sql import inspect
 from sql.util import sanitize_identifier
 from sql.cmd.cmd_utils import CmdParser
+from sql.util import expand_args, is_rendering_required
 
 
-def columns(others):
+def columns(others, user_ns):
     """
     Implementation of `%sqlcmd columns`
     This function takes in a string containing command line arguments,
     parses them to extract the name of the table and the schema, and returns
-    a list of columns for the specified table.
+    a list of columns for the specified table. It also uses the kernel
+    namespace for expanding arguments declared as variables.
 
     Parameters
     ----------
     others : str,
         A string containing the command line arguments.
 
+    user_ns : dict,
+        User namespace of IPython kernel
+
     Returns
     -------
     columns: list
@@ -26,4 +31,8 @@ def columns(others):
     parser.add_argument("-s", "--schema", type=str, help="Schema name", required=False)
 
     args = parser.parse_args(others)
+
+    if is_rendering_required(" ".join(others)):
+        expand_args(args, user_ns)
+
     return inspect.get_columns(name=sanitize_identifier(args.table), schema=args.schema)

src/sql/cmd/explore.py

@@ -7,7 +7,8 @@ def explore(others):
     Implementation of `%sqlcmd explore`
     This function takes in a string containing command line arguments,
     parses them to extract the name of the table, and displays an interactive
-    widget for exploring the contents of the specified table.
+    widget for exploring the contents of the specified table. It also uses the
+    kernel namespace for expanding arguments declared as variables.
 
     Parameters
     ----------

src/sql/cmd/profile.py

@@ -9,6 +9,7 @@ def profile(others):
     parses them to extract the name of the table, the schema, and the output location.
     It then retrieves statistical information about the specified table and either
     returns the report or writes it to the specified location.
+    It also uses the kernel namespace for expanding arguments declared as variables.
 
 
     Parameters

src/sql/cmd/snippets.py

@@ -37,7 +37,8 @@ def snippets(others):
     Implementation of `%sqlcmd snippets`
     This function handles all the arguments related to %sqlcmd snippets, namely
     listing stored snippets, and delete/ force delete/ force delete a snippet and
-    all its dependent snippets.
+    all its dependent snippets. It also uses the kernel  namespace for expanding
+    arguments declared as variables.
 
 
     Parameters

src/sql/cmd/tables.py

@@ -9,6 +9,7 @@ def tables(others):
     This function takes in a string containing command line arguments,
     parses them to extract the schema name, and returns a list of table names
     present in the specified schema or in the default schema if none is specified.
+    It also uses the kernel namespace for expanding arguments declared as variables.
 
     Parameters
     ----------

src/sql/cmd/test.py

@@ -84,6 +84,8 @@ def test(others):
     This function takes in a string containing command line arguments,
     parses them to extract the table name, column name, and conditions
     to return if those conditions are satisfied in that table
+    It also uses the kernel namespace for expanding arguments declared as
+    variables.
 
     Parameters
     ----------

src/sql/command.py

@@ -90,6 +90,9 @@ def __init__(self, magic, user_ns, line, cell) -> None:
             self.parsed["connection"] = self.args.line[0]
 
         if self.args.with_:
+            self.args.with_ = [
+                Template(item).render(user_ns) for item in self.args.with_
+            ]
             final = store.render(self.parsed["sql"], with_=self.args.with_)
             self.parsed["sql"] = str(final)
 

src/sql/magic.py

@@ -40,7 +40,6 @@
 from sql.magic_cmd import SqlCmdMagic
 from sql._patch import patch_ipython_usage_error
 from sql import util
-from sql.util import pretty_print
 from sql.error_handler import handle_exception
 from sql._current import _set_sql_magic
 
@@ -409,6 +408,9 @@ def interactive_execute_wrapper(**kwargs):
 
         args = command.args
 
+        if util.is_rendering_required(line):
+            util.expand_args(args, user_ns)
+
         if args.section and args.alias:
             raise exceptions.UsageError(
                 "Cannot use --section with --alias since the section name "
@@ -435,7 +437,7 @@ def interactive_execute_wrapper(**kwargs):
                         command.set_sql_with(with_)
                         display.message(
                             f"Generating CTE with stored snippets: \
-{pretty_print(with_)}"
+{util.pretty_print(with_)}"
                         )
                 else:
                     with_ = None

src/sql/magic_plot.py

@@ -1,4 +1,4 @@
-from IPython.core.magic import Magics, line_magic, magics_class
+from IPython.core.magic import Magics, line_magic, magics_class, no_var_expand
 from IPython.core.magic_arguments import argument, magic_arguments
 from ploomber_core.exceptions import modify_exceptions
 
@@ -21,6 +21,7 @@
 class SqlPlotMagic(Magics, Configurable):
     """%sqlplot magic"""
 
+    @no_var_expand
     @line_magic("sqlplot")
     @magic_arguments()
     @argument(
@@ -83,6 +84,9 @@ def execute(self, line="", cell="", local_ns=None):
 
         cmd = SQLPlotCommand(self, line)
 
+        if util.is_rendering_required(line):
+            util.expand_args(cmd.args, self.shell.user_ns.copy())
+
         if len(cmd.args.column) == 1:
             column = cmd.args.column[0]
         else: