loading from .ini files API changes (#803)

* renames connection_from_dsn_section -> connection_str_from_dsn_section

* adds missing docstrings

* adds missing unit tests

* adds new KeyError exception

* better errors when loading .ini file

* better error handling

* fix test

* setting the connection alias automatically when using --section

* deprecates starting connections with %sql [section_name]

* update documentation

* update changelog

* fix docs

* fix

* clean try except

* handling error when ini file is corrupted

.gitignore

@@ -1,3 +1,6 @@
+# db connection files
+*.ini
+
 # profiling data
 *.lprof
 

CHANGELOG.md

@@ -14,6 +14,8 @@
 * [Fix] Current connection and switching connections message only displayed when `feedback>=1`
 * [Fix] `--persist/--persist-replace` perform `ROLLBACK` automatically when needed
 * [Fix] `ResultSet` footer (when `displaylimit` truncates results and when showing how to convert to a data frame) now appears in the `ResultSet` plain text representation (#682)
+* [API Change] When loading connections from a `.ini` file via `%sql --section section_name`, the section name is set as the connection alias
+* [API Change] Starting connections from a `.ini` file via `%sql [section_name]` has been deprecated
 * [Doc] Fixes documentation inaccuracy that said `:variable` was deprecated (we brought it back in `0.9.0`)
 
 ## 0.9.1 (2023-08-10)

doc/_toc.yml

@@ -13,6 +13,7 @@ parts:
     - file: user-guide/tables-columns
     - file: user-guide/ggplot
     - file: user-guide/template
+    - file: user-guide/connection-file
     - file: user-guide/table_explorer
     - file: user-guide/data-profiling
 

doc/api/configuration.md

@@ -7,7 +7,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.14.7
+    jupytext_version: 1.15.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python

doc/api/magic-sql.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.14.6
+    jupytext_version: 1.15.0
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -33,7 +33,7 @@ You can view the documentation and command line arguments by running `%sql?`
     Specify creator function for new connection ([example](#specify-creator-function))
 
 ``-s`` / ``--section <section-name>``
-    Section of dsn_file to be used for generating a connection string
+    Section of dsn_file to be used for generating a connection string ([example](#start-a-connection-from-ini-file))
 
 ``-p`` / ``--persist``
     Create a table name in the database from the named DataFrame ([example](#create-table))
@@ -163,6 +163,31 @@ def creator():
 %sql --creator creator
 ```
 
+## Start a connection from `.ini file`
+
+Use `--section` to start a connection from the `dsn_filename`. To learn more, see: [](../user-guide/connection-file.md)
+
+```{code-cell} ipython3
+%config SqlMagic.dsn_filename
+```
+
+```{code-cell} ipython3
+from pathlib import Path
+
+Path("odbc.ini").write_text("""
+[duck]
+drivername = duckdb
+""")
+```
+
+```{code-cell} ipython3
+%sql --section duck
+```
+
+```{code-cell} ipython3
+%sql --connections
+```
+
 ## Create table
 
 ```{code-cell} ipython3

doc/community/vs.md

@@ -8,6 +8,8 @@ If you're migrating from `ipython-sql` to JupySQL, these are the differences (in
 
 - Since `0.6` JupySQL no longer supports old versions of IPython
 - Variable expansion is replaced from `{variable}`, `${variable}` to `{{variable}}`
+- Variable expansion via `:variable` has been disable by default, but can be enabled with [`%config SqlMagic.named_parameters = True`](../api/configuration)
+- Since `0.10.0`, loading connections from a `.ini` file using `%sql [section_name]` has been deprecated. Use `%sql --section section_name` instead.
 
 ## New features
 
@@ -16,4 +18,4 @@ If you're migrating from `ipython-sql` to JupySQL, these are the differences (in
 - Using `%sqlcmd tables` and `%sqlcmd columns --table/-t` user can quickly explore tables in the database and the columns each table has. [Click here](../user-guide/tables-columns) to learn more.
 - [Polars Integration](../integrations/polars) to convert query results to `polars.DataFrame`. `%config SqlMagic.autopolars` can be used to automatically return Polars DataFrames instead of regular result sets.
 - Integration tests with PostgreSQL, MariaDB, MySQL, SQLite and DuckDB.
-- The configuration default value of SqlMagic.displaylimit is different, in JupySQL is `10`, whereas in ipython-sql is `None`
+- The configuration default value of SqlMagic.displaylimit is different, in JupySQL is `10`, whereas in ipython-sql is `None`

doc/user-guide/connection-file.md

@@ -0,0 +1,166 @@
+---
+jupytext:
+  notebook_metadata_filter: myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.15.0
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+myst:
+  html_meta:
+    description lang=en: Using a connection file
+    keywords: jupyter, jupysql, sqlalchemy
+    property=og:locale: en_US
+---
+
+# Using a connection file
+
+Using a connection file is the recommended way to manage connections, it helps you to:
+
+- Avoid storing your credentials in your notebook
+- Manage multiple database connections
+- Define them in a single place to use it in all your notebooks
+
+```{code-cell} ipython3
+%load_ext sql
+```
+
+By default, connections are read/stored in a `odbc.ini` file:
+
+```{code-cell} ipython3
+%config SqlMagic.dsn_filename
+```
+
+However, you can change this:
+
+```{code-cell} ipython3
+%config SqlMagic.dsn_filename = "my-connections.ini"
+```
+
+The `.ini` format defines sections and you can define key-value pairs within each section. For example:
+
+```ini
+[section_name]
+key = value
+```
+
+Add a section and set the key-value pairs to add a new connection. When JupySQL loads them, it'll initialize a [`sqlalchemy.engine.URL`](https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.engine.URL.create) object and then start the connection. Valid keys are:
+
+- `drivername`: the name of the database backend
+- `username`: the username
+- `password`: database password
+- `host`: name of the host
+- `port`: the port number
+- `database`: the database name
+- `query`: a dictionary of string keys to be passed to the connection upon connect (learn more [here](https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.engine.URL.create))
+
+For example, to configure an in-memory DuckDB database:
+
+```ini
+[duck]
+drivername = duckdb
+```
+
+```{code-cell} ipython3
+from pathlib import Path
+
+_ = Path("my-connections.ini").write_text("""
+[duck]
+drivername = duckdb
+""")
+```
+
+To connect to a database defined in the connections file, use `--section` and pass the section name:
+
+```{code-cell} ipython3
+%sql --section duck
+```
+
+```{versionchanged} 0.10.0
+The connection alias is automatically set when using `%sql --section`
+```
+
+Note that the alias is set to the section name:
+
+```{code-cell} ipython3
+%sql --connections
+```
+
+```{versionchanged} 0.10.0
+Loading connections from the `.ini` (`%sql [section_name]`) file has been deprecated. Use `%sql --section section_name` instead.
+```
+
+```{code-cell} ipython3
+from urllib.request import urlretrieve
+from pathlib import Path
+
+url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
+
+if not Path("penguins.csv").exists():
+    urlretrieve(url, "penguins.csv")
+```
+
+```{code-cell} ipython3
+%%sql
+drop table if exists penguins;
+
+create table penguins as
+select * from penguins.csv
+```
+
+```{code-cell} ipython3
+%%sql
+select * from penguins
+```
+
+Let's now define another connection so we can show how we can manage multiple ones:
+
+```{code-cell} ipython3
+_ = Path("my-connections.ini").write_text("""
+[duck]
+drivername = duckdb
+
+[second_duck]
+drivername = duckdb
+""")
+```
+
+Start a new connection from the `second_duck` section name:
+
+```{code-cell} ipython3
+%sql --section second_duck
+```
+
+```{code-cell} ipython3
+%sql --connections
+```
+
+There are no tables since this is a new database:
+
+```{code-cell} ipython3
+%sqlcmd tables
+```
+
+If we switch to the first connection (by passing the alias), we'll see the table:
+
+```{code-cell} ipython3
+%sql duck
+```
+
+```{code-cell} ipython3
+%sqlcmd tables
+```
+
+We can change back to the other connection:
+
+```{code-cell} ipython3
+%sql second_duck
+```
+
+```{code-cell} ipython3
+%sqlcmd tables
+```

src/sql/connection/connection.py

@@ -161,6 +161,15 @@ def rough_dict_get(dct, sought, default=None):
     return default
 
 
+def _error_invalid_connection_info(e, connect_str):
+    err = UsageError(
+        "An error happened while creating the connection: "
+        f"{e}.{_suggest_fix(env_var=False, connect_str=connect_str)}"
+    )
+    err.modify_exception = True
+    return err
+
+
 class ConnectionManager:
     """A class to manage and create database connections"""
 
@@ -371,7 +380,7 @@ def from_connect_str(
                 )
             ) from e
         except Exception as e:
-            raise cls._error_invalid_connection_info(e, connect_str) from e
+            raise _error_invalid_connection_info(e, connect_str) from e
 
         connection = SQLAlchemyConnection(engine, alias=alias, config=config)
         connection.connect_args = connect_args
@@ -864,16 +873,7 @@ def _start_sqlalchemy_connection(cls, engine, connect_str):
             else:
                 print(e)
         except Exception as e:
-            raise cls._error_invalid_connection_info(e, connect_str) from e
-
-    @classmethod
-    def _error_invalid_connection_info(cls, e, connect_str):
-        err = UsageError(
-            "An error happened while creating the connection: "
-            f"{e}.{_suggest_fix(env_var=False, connect_str=connect_str)}"
-        )
-        err.modify_exception = True
-        return err
+            raise _error_invalid_connection_info(e, connect_str) from e
 
     def to_table(self, table_name, data_frame, if_exists, index):
         """Create a table from a pandas DataFrame"""

src/sql/exceptions.py

@@ -38,6 +38,7 @@ def _error(message):
 TypeError = exception_factory("TypeError")
 RuntimeError = exception_factory("RuntimeError")
 ValueError = exception_factory("ValueError")
+KeyError = exception_factory("KeyError")
 FileNotFoundError = exception_factory("FileNotFoundError")
 NotImplementedError = exception_factory("NotImplementedError")
 

src/sql/magic.py

@@ -422,6 +422,12 @@ def interactive_execute_wrapper(**kwargs):
 
         args = command.args
 
+        if args.section and args.alias:
+            raise exceptions.UsageError(
+                "Cannot use --section with --alias since the section name "
+                "is automatically set as the connection alias"
+            )
+
         is_cte = command.sql_original.strip().lower().startswith("with ")
 
         # only expand CTE if this is not a CTE itself
@@ -467,7 +473,7 @@ def interactive_execute_wrapper(**kwargs):
         connect_arg = command.connection
 
         if args.section:
-            connect_arg = sql.parse.connection_from_dsn_section(args.section, self)
+            connect_arg = sql.parse.connection_str_from_dsn_section(args.section, self)
 
         if args.connection_arguments:
             try:
@@ -494,7 +500,7 @@ def interactive_execute_wrapper(**kwargs):
             displaycon=self.displaycon,
             connect_args=args.connection_arguments,
             creator=args.creator,
-            alias=args.alias,
+            alias=args.section if args.section else args.alias,
             config=self,
         )
         payload["connection_info"] = conn._get_database_information()