Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polars.read_database can not work with duckdb_engine connection. #18263

Closed
2 tasks done
specter119 opened this issue Aug 19, 2024 · 1 comment · Fixed by #18277
Closed
2 tasks done

polars.read_database can not work with duckdb_engine connection. #18263

specter119 opened this issue Aug 19, 2024 · 1 comment · Fixed by #18277
Assignees
Labels
A-io-database Area: reading/writing to databases bug Something isn't working python Related to Python Polars

Comments

@specter119
Copy link

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import duckdb
import polars as pl
import sqlalchemy
from sqlalchemy import create_engine, text

duckdb.__version__, pl.__version__, sqlalchemy.__version__, duckdb_engine.__version__
# ('1.0.0', '1.5.0', '2.0.32', '0.13.1')

duckdb.sql('select 42;')
#┌───────┐
#│  42   │
#│ int32 │
#├───────┤
#│    42 │
#└───────┘

engine = create_engine('duckdb:///:memory:')
engine.connect().execute(text('SELECT 42')).fetchall()
#[(42,)]

pl.read_database('SELECT 42;', connection=engine.connect())

Log output

{
	"name": "AttributeError",
	"message": "'duckdb.duckdb.DuckDBPyConnection' object has no attribute 'c'",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 6
      3 engine = create_engine('duckdb:///:memory:')
      4 display(engine.connect().execute(text('SELECT 42')).fetchall())
----> 6 pl.read_database('SELECT 42;', connection=engine.connect())

File /opt/python3.12/lib/python3.12/site-packages/polars/io/database/functions.py:253, in read_database(query, connection, iter_batches, batch_size, schema_overrides, infer_schema_length, execute_options)
    250         raise ValueError(msg)
    252 # return frame from arbitrary connections using the executor abstraction
--> 253 with ConnectionExecutor(connection) as cx:
    254     return cx.execute(
    255         query=query,
    256         options=execute_options,
   (...)
    261         infer_schema_length=infer_schema_length,
    262     )

File /opt/python3.12/lib/python3.12/site-packages/polars/io/database/_executor.py:94, in ConnectionExecutor.__init__(self, connection)
     91 if self.driver_name == \"surrealdb\":
     92     connection = SurrealDBCursorProxy(client=connection)
---> 94 self.cursor = self._normalise_cursor(connection)
     95 self.result: Any = None

File /opt/python3.12/lib/python3.12/site-packages/polars/io/database/_executor.py:387, in ConnectionExecutor._normalise_cursor(self, conn)
    385 elif conn.engine.driver == \"duckdb_engine\":
    386     self.driver_name = \"duckdb\"
--> 387     return conn.engine.raw_connection().driver_connection.c
    388 elif self._is_alchemy_engine(conn):
    389     # note: if we create it, we can close it
    390     self.can_close_cursor = True

File /opt/python3.12/lib/python3.12/site-packages/duckdb_engine/__init__.py:95, in ConnectionWrapper.__getattr__(self, name)
     94 def __getattr__(self, name: str) -> Any:
---> 95     return getattr(self.__c, name)

AttributeError: 'duckdb.duckdb.DuckDBPyConnection' object has no attribute 'c'"
}

Issue description

read_database can not working properly

Expected behavior

return the unit cell table of 42

change https://github.com/pola-rs/polars/blob/main/py-polars/polars/io/database/_executor.py#L387
as: return conn.engine.raw_connection().driver_connection._ConnectionWrapper__c can fixed, but not sure the impact (from which version duckdb-engine change the behivor)

Installed versions

--------Version info---------
Polars:               1.5.0
Index type:           UInt32
Platform:             Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.34
Python:               3.12.5 | packaged by conda-forge | (main, Aug  8 2024, 18:36:51) [GCC 12.4.0]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2024.6.1
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           3.9.2
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             3.1.5
pandas:               2.2.2
pyarrow:              17.0.0
pydantic:             2.8.2
pyiceberg:            <not installed>
sqlalchemy:           2.0.32
torch:                2.3.1.post300
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

@specter119 specter119 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Aug 19, 2024
@alexander-beedie alexander-beedie self-assigned this Aug 20, 2024
@alexander-beedie alexander-beedie added A-io-database Area: reading/writing to databases and removed needs triage Awaiting prioritization by a maintainer labels Aug 20, 2024
@alexander-beedie
Copy link
Collaborator

Thanks for the heads-up; looks like a very simple fix on our side ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io-database Area: reading/writing to databases bug Something isn't working python Related to Python Polars
Projects
None yet
2 participants