feat: add databricks support

[ruiyang2015]

Which new backend would you like to see in Ibis?

I would like to see databricks backend support.

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

@ruiyang2015 Can you clarify a bit what "databricks" means here? Is that databricks cloud, databricks connect, or something else?

[ruiyang2015]

@ruiyang2015 Can you clarify a bit what "databricks" means here? Is that databricks cloud, databricks connect, or something else?

We use databricks connect and databricks SQL endpoints.

[Kilo59]

https://docs.databricks.com/en/sql/language-manual/index.html

https://docs.databricks.com/en/dev-tools/python-sql-connector.html

[cpcloud]

@Kilo59 Thanks for the links! The databricks DB-API looks pretty solid. They even support fetching query results as arrow tables.

[techdebtcreator]

+1 for this since my company mainly uses Databricks SQL Warehouse via the databricks-sql-connector package. Unless I missed something, I'm currently only able to connect to Databricks clusters (through the Ibis PySpark backend in conjunction with the databricks-connect package).

[nrlugg]

For reference, there is also the databricks-sdk package which could also be used for querying Databricks SQL Warehouses using the statement_execution submodule.

This package is particularly interesting because, if you use format=Format.ARROW_STREAM and disposition=Disposition.EXTERNAL_LINKS, it allows streaming chunks of serialized arrow tables (i.e., arrow IPC format), and this could potentially be used for processing tables which are larger than memory and/or be read using async or multi-threading to stream the data faster.

Also, the results of the executed query are stored temporarily in cloud storage which means the urls of executed query chunks could be cached and reused without having to execute the query again (if the query doesn't change).

Also also, pyarrow is not pinned to any particular version (unlike databricks-sql-connector where it is) which makes dependency management a less restrictive.

[hershelm]

+1 would be great to see native support for databricks

anyone in the thread, may find this blog post useful: https://posit.co/blog/databases-with-posit/

see the section on "Databricks" and the "Python" tab

[cpcloud]

We will be tackling this for the 10.0 release, stay tuned!