feat: RisingWave backend #8038
Labels
Comments
lostmygithubaccount
added
feature
This was referenced Jan 19, 2024
Closed
cpcloud
pushed a commit
that referenced
this issue
Jan 28, 2024
## Description of changes This is the initial PR for ibis to support risingwave. [RisingWave](https://github.com/risingwavelabs/risingwave) is a distributed SQL streaming database engineered to provide the simplest and most cost-efficient approach for processing and managing streaming data with utmost reliability. After a few weeks of investigation, here's my phasic results. 1. As Risingwave is largely compatible with Postgres, Ibis can be easily extended to support Risingwave. The main work is to add a new dialect for Risingwave, which is similar to the `postgres` dialect. I've almost finished this part in this PR. With this PR, Ibis can be used to connect to Risingwave and run some basic queries. I've manually tested some queries which works well and add some tests imitating Postgres Backend. I would appreciate if you can help test more queries. 2. As ibis relies on SQLalchemy to support Postgres, we follow its implementation to support Risingwave. However, there are also some differences in semantics between Risingwave and Postgres, which require some modification in either ibis or SQLalchemy. `sqlalchemy-risingwave` is designed to reduce this mismatch. So in this PR, I introduce the new dependencies `sqlalchemy-risingwave` to ibis. 3. Ibis has no support for Materialized View natively. However Materialized View is a core concept in RW, people use RW because of its convenient auto-updated Materialized View. Now, if a user wants to create a new MV, he needs to use a raw SQL. Adding DDLs like `CreateMaterializedView`, `CreateSource` and `CreateSink` in the [base ddl file](https://github.com/ibis-project/ibis/blob/main/ibis/backends/base/sql/ddl.py) may help. We would appreciate it if you can help offer some suggestions. Besides this, I also met some obstacle that may need your help. 1. Risingwave hasn't supported `TEMPORARY VIEW` yet, so I changed some implementations relying on `TEMPORARY VIEW` to a normal view. For example, for the `_metadata()` functions, RW backend's implementation is `con.exec_driver_sql(f"CREATE VIEW IF NOT EXISTS {name} AS {query}")`. While in PG it's `con.exec_driver_sql(f"CREATE TEMPORARY VIEW {name} AS {query}")`. Do you have any suggestions for other ways to work around this? Besides, I didn't quite get what `_metadata()` is doing here. I would appreciate it if you could explain it a bit. 2. There's some mismatch between the `postgres` dialect and `risingwave` dialect, which are still not fully tested in this PR. We'll continue to work on it. 3. ~~This PR requires some new features of Risingwave v1.6.0 and sqlalchemy-risingwave v1.0.0 which are not released yet. They'll be released soon.~~ Done. BTW, how should I indicate this backend is only for risingwave > 1.6? 4. I don't quite understand the test pipeline of Ibis. I copied the test cases from the `postgres` dialect and modified them to fit the `risingwave` dialect, and some of them are commented temporarily due to the lack of support. I also added an SQL script to help set up the test environment, which creates tables and loads data. But I don't know how to run it in the test pipeline. Any suggestions or guidance are welcomed. I suppose the test pipeline would require a docker image of Risingwave. We can provide one if needed. 5. I'm a newbie in the ibis community, this PR may not be perfect considering others. Any comments are welcomed and I sincerely appreciate your time and patience. closes #8038
github-project-automation
bot
moved this from backlog
to done
in Ibis planning and roadmap
gforsyth
pushed a commit
to gforsyth/ibis
that referenced
this issue
Jan 31, 2024
This is the initial PR for ibis to support risingwave. [RisingWave](https://github.com/risingwavelabs/risingwave) is a distributed SQL streaming database engineered to provide the simplest and most cost-efficient approach for processing and managing streaming data with utmost reliability. After a few weeks of investigation, here's my phasic results. 1. As Risingwave is largely compatible with Postgres, Ibis can be easily extended to support Risingwave. The main work is to add a new dialect for Risingwave, which is similar to the `postgres` dialect. I've almost finished this part in this PR. With this PR, Ibis can be used to connect to Risingwave and run some basic queries. I've manually tested some queries which works well and add some tests imitating Postgres Backend. I would appreciate if you can help test more queries. 2. As ibis relies on SQLalchemy to support Postgres, we follow its implementation to support Risingwave. However, there are also some differences in semantics between Risingwave and Postgres, which require some modification in either ibis or SQLalchemy. `sqlalchemy-risingwave` is designed to reduce this mismatch. So in this PR, I introduce the new dependencies `sqlalchemy-risingwave` to ibis. 3. Ibis has no support for Materialized View natively. However Materialized View is a core concept in RW, people use RW because of its convenient auto-updated Materialized View. Now, if a user wants to create a new MV, he needs to use a raw SQL. Adding DDLs like `CreateMaterializedView`, `CreateSource` and `CreateSink` in the [base ddl file](https://github.com/ibis-project/ibis/blob/main/ibis/backends/base/sql/ddl.py) may help. We would appreciate it if you can help offer some suggestions. Besides this, I also met some obstacle that may need your help. 1. Risingwave hasn't supported `TEMPORARY VIEW` yet, so I changed some implementations relying on `TEMPORARY VIEW` to a normal view. For example, for the `_metadata()` functions, RW backend's implementation is `con.exec_driver_sql(f"CREATE VIEW IF NOT EXISTS {name} AS {query}")`. While in PG it's `con.exec_driver_sql(f"CREATE TEMPORARY VIEW {name} AS {query}")`. Do you have any suggestions for other ways to work around this? Besides, I didn't quite get what `_metadata()` is doing here. I would appreciate it if you could explain it a bit. 2. There's some mismatch between the `postgres` dialect and `risingwave` dialect, which are still not fully tested in this PR. We'll continue to work on it. 3. ~~This PR requires some new features of Risingwave v1.6.0 and sqlalchemy-risingwave v1.0.0 which are not released yet. They'll be released soon.~~ Done. BTW, how should I indicate this backend is only for risingwave > 1.6? 4. I don't quite understand the test pipeline of Ibis. I copied the test cases from the `postgres` dialect and modified them to fit the `risingwave` dialect, and some of them are commented temporarily due to the lack of support. I also added an SQL script to help set up the test environment, which creates tables and loads data. But I don't know how to run it in the test pipeline. Any suggestions or guidance are welcomed. I suppose the test pipeline would require a docker image of Risingwave. We can provide one if needed. 5. I'm a newbie in the ibis community, this PR may not be perfect considering others. Any comments are welcomed and I sincerely appreciate your time and patience. closes ibis-project#8038
cpcloud
pushed a commit
to gforsyth/ibis
that referenced
this issue
Feb 1, 2024
This is the initial PR for ibis to support risingwave. [RisingWave](https://github.com/risingwavelabs/risingwave) is a distributed SQL streaming database engineered to provide the simplest and most cost-efficient approach for processing and managing streaming data with utmost reliability. After a few weeks of investigation, here's my phasic results. 1. As Risingwave is largely compatible with Postgres, Ibis can be easily extended to support Risingwave. The main work is to add a new dialect for Risingwave, which is similar to the `postgres` dialect. I've almost finished this part in this PR. With this PR, Ibis can be used to connect to Risingwave and run some basic queries. I've manually tested some queries which works well and add some tests imitating Postgres Backend. I would appreciate if you can help test more queries. 2. As ibis relies on SQLalchemy to support Postgres, we follow its implementation to support Risingwave. However, there are also some differences in semantics between Risingwave and Postgres, which require some modification in either ibis or SQLalchemy. `sqlalchemy-risingwave` is designed to reduce this mismatch. So in this PR, I introduce the new dependencies `sqlalchemy-risingwave` to ibis. 3. Ibis has no support for Materialized View natively. However Materialized View is a core concept in RW, people use RW because of its convenient auto-updated Materialized View. Now, if a user wants to create a new MV, he needs to use a raw SQL. Adding DDLs like `CreateMaterializedView`, `CreateSource` and `CreateSink` in the [base ddl file](https://github.com/ibis-project/ibis/blob/main/ibis/backends/base/sql/ddl.py) may help. We would appreciate it if you can help offer some suggestions. Besides this, I also met some obstacle that may need your help. 1. Risingwave hasn't supported `TEMPORARY VIEW` yet, so I changed some implementations relying on `TEMPORARY VIEW` to a normal view. For example, for the `_metadata()` functions, RW backend's implementation is `con.exec_driver_sql(f"CREATE VIEW IF NOT EXISTS {name} AS {query}")`. While in PG it's `con.exec_driver_sql(f"CREATE TEMPORARY VIEW {name} AS {query}")`. Do you have any suggestions for other ways to work around this? Besides, I didn't quite get what `_metadata()` is doing here. I would appreciate it if you could explain it a bit. 2. There's some mismatch between the `postgres` dialect and `risingwave` dialect, which are still not fully tested in this PR. We'll continue to work on it. 3. ~~This PR requires some new features of Risingwave v1.6.0 and sqlalchemy-risingwave v1.0.0 which are not released yet. They'll be released soon.~~ Done. BTW, how should I indicate this backend is only for risingwave > 1.6? 4. I don't quite understand the test pipeline of Ibis. I copied the test cases from the `postgres` dialect and modified them to fit the `risingwave` dialect, and some of them are commented temporarily due to the lack of support. I also added an SQL script to help set up the test environment, which creates tables and loads data. But I don't know how to run it in the test pipeline. Any suggestions or guidance are welcomed. I suppose the test pipeline would require a docker image of Risingwave. We can provide one if needed. 5. I'm a newbie in the ibis community, this PR may not be perfect considering others. Any comments are welcomed and I sincerely appreciate your time and patience. closes ibis-project#8038
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem?
opening for tracking purposes, work in progress in #7954
Describe the solution you'd like
a RisingWave backend to use Ibis for streaming data w/ RisingWave, a postgres-compatible streaming data platform
What version of ibis are you running?
n/a
What backend(s) are you using, if any?
n/a
Code of Conduct
The text was updated successfully, but these errors were encountered: