Remote table implementation #190
Conversation
Namely, make sure to run all queries in a blocking task, since that is what the underlying (postgres) driver is trying to do.
tests/end_to_end.rs
Outdated
| (1, 1.1, 'one', '2022-11-01'),\ | ||
| (2, 2.22, 'two', '2022-11-02'),\ | ||
| (3, 3.333, 'three', '2022-11-03'),\ | ||
| (4, 4.4444, 'four', '2022-11-04')", |
There was a problem hiding this comment.
Are there any gnarly types that we expect to work / not work? I'm thinking PG arrays or JSON fields. Would be nice to be able to recover them, at least as strings.
There was a problem hiding this comment.
Yeah, arrays and JSON fields don't work because of the limitations of connectorx (in particular no conversion rules for the arrow destination for those fields).
In addition I've found that TIMESTAMPZ field conversion is broken in connectorx—despite there being an adequate conversion definition, the actual implementation is not there yet.
src/remote_tables.rs
Outdated
| if let Some(indices) = projection { | ||
| schema = schema.project(indices)?; | ||
| columns = schema | ||
| .fields() | ||
| .iter() | ||
| .map(|f| f.name().as_str()) | ||
| .collect::<Vec<&str>>() | ||
| .map(|f| format!("\"{}\"", f.name())) |
There was a problem hiding this comment.
It should be something like f.name().replace("\"", "\"\"") (at least in PG, you can escape double quotes in identifier names with double-double quotes, though in MySQL identifiers are surrounded by backticks, so this won't be platform-independent.
This PR introduces a new remote table concept, analogous to FDWs in Postgres, that uses the connector-x lib to query a supported remote database.
Things missing in the implementation, that could/should be done separately (some of which are related):
sqlx = "^0.6.2"(needs^0.24.1version oflibsqlite3-sys) andconnectorx = "^0.3.1"(needslibsqlite3-sys = "^0.22.2").datafusion-remote-tables.