feat: add datafusion storage catalog #1381
Conversation
roeap
requested review from
houqp,
xianwill,
wjones127,
fvaleye,
rtyler and
mosyp
as code owners
| /// Underlying object store | ||
| store: Arc<dyn ObjectStore>, | ||
| /// A map of table names to a fully quilfied storage location | ||
| tables: DashMap<String, String>, |
There was a problem hiding this comment.
Is there an artifact of implementing this SchemaProvider that requires the use of DashMap? Since this adds a new dependency, I'm curious what forces this requirement since it's not clear to me
There was a problem hiding this comment.
We could also go the "classic" route using Arc<Mutex<...>>, however since datafusion internally switched in most places, we already have that as a transient dependency if we use datafusion.
It does however simplify all code where we use this.
| parent = p; | ||
| } | ||
| } | ||
| for table in tables.into_iter() { |
There was a problem hiding this comment.
This bit of logic is a bit obtuse to me, especially the series of chained calls to create file_name. I think it's a good candidate for refactoring into a private function with some unit tests, but at your discretion there since I know sometimes these objects are a PITA to scaffold for tests
Adds a datafusion `SchemaProvider` implementation based on discovering Delta tables in a storage location via `_delta_log` folders. <!--- For example: - closes #106 ---> <!--- Share links to useful documentation --->
Description
Adds a datafusion
SchemaProviderimplementation based on discovering Delta tables in a storage location via_delta_logfolders.Related Issue(s)
Documentation