Allow Spark SQL as a dialect

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## 0.10.7dev
 
+* [Feature] Add Spark Connection as a dialect for Jupysql ([#965](https://github.com/ploomber/jupysql/issues/965)) (by [@gilandose](https://github.com/gilandose))
+
 ## 0.10.6 (2023-12-21)
 
 * [Fix] Fix error when `%sql` includes a query with negative numbers ([#958](https://github.com/ploomber/jupysql/issues/958))

doc/_toc.yml

@@ -43,6 +43,7 @@ parts:
       - file: integrations/duckdb-native
       - file: integrations/compatibility
       - file: integrations/chdb
+      - file: integrations/spark
 
   - caption: API Reference
     chapters:

doc/api/configuration.md

@@ -234,6 +234,26 @@ value enables the ones from previous values plus new ones:
 - `2`: All feedback
   - Footer to distinguish pandas/polars data frames from JupySQL's result sets
 
+## `lazy_execution`
+
+```{versionadded} 0.10.7
+This option only works when connecting to Spark
+```
+
+Default: `False`
+
+Return lazy relation to dataset rather than executing through JupySql.
+
+```{code-cell} ipython3
+%config SqlMagic.lazy_execution = True
+df = %sql SELECT * FROM languages
+```
+
+```{code-cell} ipython3
+%config SqlMagic.lazy_execution = False
+res = %sql SELECT * FROM languages
+```
+
 ## `named_parameters`
 
 ```{versionadded} 0.9

doc/conf.py

@@ -27,6 +27,7 @@
     "integrations/oracle.ipynb",
     "integrations/snowflake.ipynb",
     "integrations/redshift.ipynb",
+    "integrations/spark.ipynb",
 ]
 nb_execution_in_temp = True
 nb_execution_show_tb = True

doc/integrations/compatibility.md

@@ -114,4 +114,20 @@ These table reflects the compatibility status of JupySQL `>=0.7`
 - Listing tables with `%sqlcmd tables` ✅
 - Listing columns with `%sqlcmd columns` ✅
 - Parametrized SQL queries via `{{parameter}}` ✅
-- Interactive SQL queries via `--interact` ✅
+- Interactive SQL queries via `--interact` ✅
+
+## Spark
+
+- Running queries with `%%sql` ✅
+- CTEs with `%%sql --save NAME` ✅
+- Plotting with `%%sqlplot boxplot` ❓
+- Plotting with `%%sqlplot bar` ✅
+- Plotting with `%%sqlplot pie` ✅
+- Plotting with `%%sqlplot histogram` ✅
+- Plotting with `ggplot` ✅ 
+- Profiling tables with `%sqlcmd profile` ✅
+- Listing tables with `%sqlcmd tables` ❌
+- Listing columns with `%sqlcmd columns` ❌
+- Parametrized SQL queries via `{{parameter}}` ✅
+- Interactive SQL queries via `--interact` ✅
+- Persisting Dataframes via `--persist` ✅