meta: Release and launch Ibis streaming data backends

[lostmygithubaccount]

Acceptance criteria: Ibis releases and launches its first two streaming backends, Apache Flink and RisingWave, in collaboration with Claypot AI and RisingWave respectively.

Stretch goal: Apache Sedona backend for streaming + geospatial.

TODOs:

• feat: Apache Flink backend #8036
• feat: RisingWave backend #8038
• Apache Flink tutorial + release blog
  • docs(flink): add a basic "getting started" example #7739
• Overall "Ibis supports streaming backends" blog

separately we will track:

• docs: RisingWave blog / tutorial  #8424

[lostmygithubaccount]

Weekly Update:
2024-01-18

After the sync with Claypot on 01/16/24 these are the PRs that need review for the 8.0.0 release.

Flink
To review

(Chloe mentioned that Mehmet's PR should be reviewed first so this is the order):

• test(flink): deep dive on the tests marked for Flink in test_json.py ibis#7908
• feat(flink): implement support for temporal join ibis#7921
• feat(flink): implement array operators ibis#7951
• feat(flink): add new temporal operators ibis#7974
• test(flink): add unit test for window join ibis#7966

Draft PR: conversation ongoing in the comments on whether it's worth it keep this one open

• feat(flink): implement struct field, clean up literal, and adjust timecontext test markers ibis#7997

This is the blog:

• docs(blog): add post introducing the Flink backend ibis#7912

Risingwave
This is the open PR, Deepyaman mentioned he will help reviewing but this one needs some attention from the team, during the meeting Phillip mentioned he will take a look.

At the moment CI is stuck because it has conflicts, maybe we should give it a nudge.

• feat(risingwave): init impl for Risingwave ibis#7954

(copying over last week's update on behalf of Naty)

[ncclementi]

Weekly Update 2024/01/24

Flink
Still open, need to get in before 8.0 release

• feat(flink): implement array operators #7951 (Looks like needs a final review)
• test(flink): add unit test for window join #7966 (there is a comment pending, and a ci linting job failing)

Merged or closed 🎉

• test(flink): deep dive on the tests marked for Flink in test_json.py #7908
• feat(flink): add new temporal operators #7974
• feat(flink): implement struct field, clean up literal, and adjust timecontext test markers #7997
• feat(flink): implement support for temporal join #7921

Blog PR:

• docs(blog): add post introducing the Flink backend #7912 (There are few comments to address, but this should get merged and publicized next week )

Tutorial PR:

• docs(tutorial): add a tutorial for the Flink backend #8085 (Draft - WIP)

Risingwave

• feat(risingwave): init impl for Risingwave #7954 (CI is mostly green, only one failure on a Risingwave job)

[ncclementi]

Weekly Update

2024/01/31

NOTE: 8.0 Ibis release with streaming support will happen early next week

Flink
Still open, need to get in before 8.0 release

• feat(flink): implement array operators #7951 (Needs a final review - Philip is on it)

Still open but not required for release

• test(flink): add unit test for window join #7966 (there is a comment pending, and a ci linting job failing)

Blog PR:

• docs(blog): add post introducing the Flink backend #7912 - This one is ready, waiting for 8.0 release to be merged and publicized

Tutorial PR:

• docs(tutorial): add a tutorial for the Flink backend #8085 (Under review)

Risingwave PR was merged 🎉

• feat(risingwave): init impl for Risingwave #7954

[ncclementi]

Weekly update

2024/02/07

Flink
This PRs were merged and the Flink backend was concluded for the release of Ibis v8.0

• feat(flink): implement array operators #7951
• test(flink): add unit test for window join #7966

Blog and Tutorial PRs are ready to be merged, waiting for coordinated advertisement push:

• docs(blog): add post introducing the Flink backend #7912
• docs(tutorial): add a tutorial for the Flink backend #8085

RisingWave

• No update here

[ncclementi]

@lostmygithubaccount In the main epic comment we have these as requirements. For release and launch.

• RisingWave tutorial + release blog
• Overall "Ibis supports streaming backends" blog

Can we revisit these requirements?

I think having API docs for RisingWave (added here #8286) and the roadmap blog announcement that will have information about streaming should probably be enough for release announcement.

Thoughts?

[ncclementi]

Weekly update

2024/02/14

• Ibis streaming support was released in v8.0
• Blogs and tutorial on the Flink backend were released
• Coordinated launch and press release were done to promote the streaming feature

[ncclementi]

Weekly update

2024/02/22

• Overall "Ibis supports streaming backends" blog was covered in the Ibis roadmap blog (https://ibis-project.org/posts/roadmap-2024-h1/)
• Rising wave did a press released about the Ibis-RisingWave backend
  https://risingwave.com/blog/risingwave-labs-and-voltron-data-collaborate-to-accelerate-stream-processing/

We have pending a RisingWave blog/tutorial on Ibis site.

• docs: RisingWave blog / tutorial  #8424