Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: conceptual explanation of Ibis, ADBC, Substrait relationships #6618

Closed
lostmygithubaccount opened this issue Jul 11, 2023 · 0 comments · Fixed by #7898
Closed

docs: conceptual explanation of Ibis, ADBC, Substrait relationships #6618

lostmygithubaccount opened this issue Jul 11, 2023 · 0 comments · Fixed by #7898
Assignees
@lostmygithubaccount
Labels
docs Documentation related issues or PRs

Comments

@lostmygithubaccount
Copy link
Member

today I saw that one can use Polars to query Snowflake directly using ADBC https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_database.html

and started thinking what should be the relationship between ADBC and Ibis. I saw various closed PRs on the Ibis side (#5010, #4267) so initially this was not clear to me.

Digging a bit more, I also came across Substrait and initially it was not obvious to me what the relationship between Substrait and ADBC was. I finally found this:

ADBC allows applications to pass queries either as SQL strings or Substrait plans.

https://substrait.io/community/powered_by/

Do you have any more insight on what are the overall plans and roadmap in this space of the whole Ibis -> Backend -> Substrait -> ADBC -> Database tower?

Originally posted by @astrojuanlu in #6617
@lostmygithubaccount lostmygithubaccount mentioned this issue Aug 9, 2023
Closed
28 tasks
@lostmygithubaccount lostmygithubaccount added the docs Documentation related issues or PRs label Dec 7, 2023
@lostmygithubaccount lostmygithubaccount moved this from backlog to todo in Ibis planning and roadmap Dec 7, 2023
@lostmygithubaccount lostmygithubaccount moved this from todo to cooking in Ibis planning and roadmap Jan 4, 2024
@lostmygithubaccount lostmygithubaccount mentioned this issue Jan 4, 2024
Merged
3 tasks
@lostmygithubaccount lostmygithubaccount moved this from cooking to review in Ibis planning and roadmap Jan 5, 2024
@github-project-automation github-project-automation bot moved this from review to done in Ibis planning and roadmap Jan 5, 2024
cpcloud pushed a commit that referenced this issue Jan 5, 2024
@lostmygithubaccount
docs: add composable data ecosystem concept (#7898)
d78a887 
## Description of changes

add conceptual article for the composable data ecosystem, explaining the
relationship between Ibis and other key projects (Apache Arrow, ADBC,
Substrait)

very WIP. some TODOs:

- [x] finish prose
- [x] formatting
- [x] share w/ other projects for review/feedback/contribution

I plan to finish my first pass on this ~tomorrow and share w/ other
projects. some open questions:

- is it worth mentioning Flight and FlightSQL? I kinda feel like it's
confusing and focusing on ADBC as the connectivity layer is better
- are there other projects we should mention? perhaps SQLGlot fits in
here?
- Substrait: working through the finer points of why SQL isn't
sufficient and Substrait is needed

## Issues closed

closes #6618
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lostmygithubaccount lostmygithubaccount

Labels
docs Documentation related issues or PRs
Projects
Ibis planning and roadmap
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

docs: add composable data ecosystem concept
1 participant
@lostmygithubaccount