Make SourceConn clonable and bump Arrow/Datafusion #397
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi folks 👋🏻
We're using connector-x to build a remote table querying capability in Seafowl (an analytical, cache friendly, time travel-able database we're building on top of Datafusion).
Given that on one hand we rely on tokio async runtime, while on the other hand some of the DB drivers block the thread when executing queries, we're forced to run the queries in a closure using
spawn_blocking
.In turn, instead of creating a single
SourceConn
upon instantiating the remote table, we're forced to instantiate a newSourceConn
for each query execution, since otherwise we can't pass the ref to theget_arrow
function inside the closure due to lifetimes incompatibilities (borrowed data escapes outside of associated function
).Long story short, to mitigate this we'd really only need
SourceConn
to be cloneable (unless this is for some reason a bad idea?), as then we could get away with keeping a single one in remote table struct and clone it or it's fields when needed, hence this PR.Additionally, I bumped Arrow and Datafusion to the latest available versions (26 and 14 respectively).