Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make SourceConn clonable and bump Arrow/Datafusion #397

Closed
wants to merge 2 commits into from

Conversation

gruuya
Copy link
Contributor

@gruuya gruuya commented Nov 8, 2022

Hi folks 👋🏻

We're using connector-x to build a remote table querying capability in Seafowl (an analytical, cache friendly, time travel-able database we're building on top of Datafusion).

Given that on one hand we rely on tokio async runtime, while on the other hand some of the DB drivers block the thread when executing queries, we're forced to run the queries in a closure using spawn_blocking.

In turn, instead of creating a single SourceConn upon instantiating the remote table, we're forced to instantiate a new SourceConn for each query execution, since otherwise we can't pass the ref to the get_arrow function inside the closure due to lifetimes incompatibilities (borrowed data escapes outside of associated function).

Long story short, to mitigate this we'd really only need SourceConn to be cloneable (unless this is for some reason a bad idea?), as then we could get away with keeping a single one in remote table struct and clone it or it's fields when needed, hence this PR.

Additionally, I bumped Arrow and Datafusion to the latest available versions (26 and 14 respectively).

@alswang18
Copy link
Contributor

@wangxiaoying I think this should be fine. We are only making a string container clonable as far as I can tell for this PR. Would love to know your thoughts.

@wangxiaoying
Copy link
Contributor

wangxiaoying commented Nov 9, 2022

Hi @gruuya , thanks for the PR! To pass the CI, I made a minor change to align arrow version in rust with cpp and python crates. I'm going to close this PR and redirect to #399 (since I do not have access to the upstream).

@gruuya
Copy link
Contributor Author

gruuya commented Nov 9, 2022

Great, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants