Variable expansion outside queries

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## 0.10.8dev
 * [Fix] Fix edge case where `select` and other SQL keywords were not properly used to find where the user's query started, causing argument parsing issues (#973)
+* [Feature] Add support for parametrizing string type arguments of `%%sql`, `%sqlplot`, `%sqlcmd`' (#699)
 
 ## 0.10.7 (2023-12-23)
 

doc/_toc.yml

@@ -13,6 +13,7 @@ parts:
     - file: user-guide/tables-columns
     - file: user-guide/ggplot
     - file: user-guide/template
+    - file: user-guide/argument-expansion
     - file: user-guide/connection-file
     - file: user-guide/table_explorer
     - file: user-guide/data-profiling

doc/api/magic-plot.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.15.1
+    jupytext_version: 1.16.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -293,3 +293,50 @@ You can also show the percentage on top of the pie using the `S`/`show-numbers`
 ```{code-cell} ipython3
 %sqlplot pie --table penguins.csv --column species -S
 ```
+
+## Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+```{code-cell} ipython3
+%%sql 
+DROP TABLE IF EXISTS penguins;
+CREATE SCHEMA IF NOT EXISTS s1;
+CREATE TABLE s1.penguins (
+    species VARCHAR(255),
+    island VARCHAR(255),
+    bill_length_mm DECIMAL(5, 2),
+    bill_depth_mm DECIMAL(5, 2),
+    flipper_length_mm DECIMAL(5, 2),
+    body_mass_g INTEGER,
+    sex VARCHAR(255)
+);
+COPY s1.penguins FROM 'penguins.csv' WITH (FORMAT CSV, HEADER TRUE);
+```
+
+```{code-cell} ipython3
+table = "penguins"
+schema = "s1"
+orient = "h"
+column = "bill_length_mm"
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{table}} --schema {{schema}} --column {{column}} --orient {{orient}}
+```
+
+Now let's see another example using `--with`:
+
+```{code-cell} ipython3
+snippet = "gentoo"
+```
+
+```{code-cell} ipython3
+%%sql --save {{snippet}}
+SELECT * FROM {{schema}}.{{table}} 
+WHERE species == 'Gentoo'
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{snippet}} --with {{snippet}} --column {{column}}
+```

doc/api/magic-profile.md

@@ -201,4 +201,28 @@ Let’s profile `my_numbers` of `b_schema`
 
 ```{code-cell} ipython3
 %sqlcmd profile --table my_numbers --schema b_schema
-```
+```
+
+# Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's look at an example that uses variable expansion for `table`, `schema` and `output` arguments:
+
+```{code-cell} ipython3
+table = "my_numbers"
+schema = "b_schema"
+output = "numbers-report.html"
+```
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%sqlcmd profile --table {{table}} --schema {{schema}} --output {{output}}
+```
+
+```{code-cell} ipython3
+from IPython.display import HTML
+
+HTML(output)
+```

doc/api/magic-snippets.md

@@ -150,3 +150,28 @@ Now, force delete `chinstrap` and its dependent `chinstrap_sub`:
 ```{code-cell} ipython3
 %sqlcmd snippets -A chinstrap
 ```
+
+
+## Parameterizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's see some examples:
+
+```{code-cell} ipython3
+snippet_name = "gentoo"
+```
+
+```{code-cell} ipython3
+%%sql --save {{snippet_name}}
+SELECT * FROM penguins.csv where species == 'Gentoo'
+```
+
+```{code-cell} ipython3
+gentoo_snippet = %sqlcmd snippets {{snippet_name}}
+print(gentoo_snippet)
+```
+
+```{code-cell} ipython3
+%sqlcmd snippets -d {{snippet_name}}
+```

doc/api/magic-sql.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.15.1
+    jupytext_version: 1.16.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -359,3 +359,21 @@ LIMIT 3
 ```{code-cell} ipython3
 %sql --file my-query.sql
 ```
+
+## Parameterizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's see an example of creating a connection using an alias and closing the same through variable substitution.
+
+```{code-cell} ipython3
+alias = "db-four"
+```
+
+```{code-cell} ipython3
+%sql sqlite:///db_four.db --alias {{alias}}
+```
+
+```{code-cell} ipython3
+%sql --close {{alias}}
+```

doc/api/magic-tables-columns.md

@@ -101,6 +101,20 @@ CREATE TABLE s2.t2(id INTEGER PRIMARY KEY, j VARCHAR);
 %sqlcmd tables -s s1
 ```
 
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+Let's see an example:
+
+```{code-cell} ipython3
+schema = "s1"
+```
+
+```{code-cell} ipython3
+:tags: [hide-output]
+
+%sqlcmd tables -s {{schema}}
+```
+
 As expected, the argument returns the table names under schema s1, which is t1.
 
 +++
@@ -123,3 +137,16 @@ Arguments:
 
 %sqlcmd columns -s s1 -t t1
 ```
+
+JupySQL also supports variable expansion of arguments of `columns`. Let's see an example:
+
+```{code-cell} ipython3
+
+table = "t1"
+schema = "s1"
+```
+
+```{code-cell} ipython3
+
+%sqlcmd columns -s {{schema}} -t {{table}}
+```

doc/howto/testing-columns.md

@@ -80,3 +80,16 @@ The test fails, returning both Shakespeare and Brecht.
 
 Currently, 5 different comparator arguments are supported: `greater`, `greater-or-equal`, `less-than`, `less-than-or-equal`, and `no-nulls`. 
 
+## Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. Let's see an example of running tests using parametrization:
+
+```{code-cell} ipython3
+table = "writer"
+column = "year_of_death"
+limit = "2000"
+```
+
+```{code-cell} ipython3
+%sqlcmd test --table {{table}} --column {{column}} --less-than {{limit}}
+```

doc/user-guide/argument-expansion.md

@@ -0,0 +1,108 @@
+---
+jupytext:
+  notebook_metadata_filter: myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.14.7
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+myst:
+  html_meta:
+    description lang=en: Variable substitution of arguments in Jupyter via JupySQL
+    keywords: jupyter, sql, jupysql, jinja
+    property=og:locale: en_US
+---
+
+# Parameterizing arguments
+
+```{versionadded} 0.10.8
+JupySQL uses Jinja templates for enabling parametrization of arguments. Arguments are parametrized with `{{variable}}`.
+```
+
+
+## Parametrization via `{{variable}}`
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. This allows the user to specify arguments with placeholders that can be replaced by variables dynamically.
+
+The benefits of using parametrized arguments is that they can be reused for different purposes.
+
+Let's load some data and connect to the in-memory DuckDB instance:
+
+```{code-cell} ipython3
+%load_ext sql
+%sql duckdb://
+%config SqlMagic.displaylimit = 3
+```
+
+```{code-cell} ipython3
+filename = "penguins.csv"
+```
+
+
+```{code-cell} ipython3
+from pathlib import Path
+from urllib.request import urlretrieve
+
+if not Path("penguins.csv").is_file():
+    urlretrieve(
+        "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv",
+        filename,
+    )
+```
+
+Now let's create a snippet from the data by declaring a `table` variable and use it in the `--save` argument.
+
++++
+
+### Create a snippet
+
+```{code-cell} ipython3
+table = "penguins_data"
+```
+
+```{code-cell} ipython3
+%%sql --save {{table}}
+SELECT *
+FROM penguins.csv
+```
+
+```{code-cell} ipython3
+snippet = %sqlcmd snippets {{table}}
+print(snippet)
+```
+
+
+### Plot a histogram
+
+Now, let's declare a variable `column` and plot a histogram on the data.
+
+```{code-cell} ipython3
+column = "body_mass_g"
+```
+
+```{code-cell} ipython3
+%sqlplot boxplot --table {{table}} --column {{column}}
+```
+
+### Profile and Explore
+
+We can use the `filename` variable to profile and explore the data as well:
+
+```{code-cell} ipython3
+%sqlcmd profile --table {{filename}}
+```
+
+```{code-cell} ipython3
+%sqlcmd explore --table {{filename}}
+```
+
+### Run some tests
+
+```{code-cell} ipython3
+%sqlcmd test --table {{table}} --column {{column}} --greater 3500
+```
+

doc/user-guide/data-profiling.md

@@ -119,3 +119,23 @@ Let's profile `my_numbers` of `b_schema`
 ```{code-cell} ipython3
 %sqlcmd profile --table trips --schema some_schema
 ```
+
+### Parametrizing arguments
+
+JupySQL supports variable expansion of arguments in the form of `{{variable}}`. Let's see an example using `table`, `schema` and `output`.
+
+```{code-cell} ipython3
+table = "trips"
+schema = "some_schema"
+output = "my-report.html"
+```
+
+```{code-cell} ipython3
+%sqlcmd profile --table {{table}} --schema {{schema}} --output {{output}}
+```
+
+```{code-cell} ipython3
+from IPython.display import HTML
+
+HTML(output)
+```