Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(pyspark): support reading from and writing to Kafka #9266

Merged
merged 12 commits into from
Jun 3, 2024
Merged

feat(pyspark): support reading from and writing to Kafka #9266

merged 12 commits into from
Jun 3, 2024

Conversation

chloeh13q
Copy link
Contributor

Description of changes

Introduce read_kafka and to_kafka methods to support reading from and writing to Kafka in streaming mode.

@chloeh13q chloeh13q marked this pull request as ready for review May 30, 2024 14:52
@chloeh13q chloeh13q requested a review from jcrist May 30, 2024 15:08
@chloeh13q chloeh13q mentioned this pull request May 30, 2024
Closed
1 task
cpcloud
cpcloud requested changes Jun 3, 2024
View reviewed changes
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few changes are needed here.

ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
ibis/backends/pyspark/__init__.py Outdated Show resolved Hide resolved
Chloe He added 11 commits June 3, 2024 08:43
feat(pyspark): provide a mode option to manage both batch and streami…
dd8d30c 
…ng connections
feat(pyspark): implement read_kafka method
a85b0fd
feat(pyspark): implement read and write methods and unit tests
7114b7d
refactor: move test data to ibis-testing-data, fix a small bug
c583ecd
refactor(pyspark): rename read and write methods to reflect directory…
ee2e6eb 
… level read/write and update ibis-test-data directory structure
test(pyspark): add tests for read and write in pyspark streaming
8bc6e2c
refactor(pyflink): consolidate methods into a single Backend class
b884d67
fix a few minor things
2f4e7db
address comments
10f5400
feat(pyspark): implement new methods to support reading from and writ…
7b75b5b 
…ing to kafka
make the write method return the query object
0b07233
@chloeh13q chloeh13q requested a review from cpcloud June 3, 2024 17:02
cpcloud
cpcloud approved these changes Jun 3, 2024
View reviewed changes
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Curious to see how this gets used!

@cpcloud cpcloud added this to the 9.1 milestone Jun 3, 2024
@cpcloud cpcloud added pyspark The Apache PySpark backend streaming Issue related to streaming APIs or backends labels Jun 3, 2024
@cpcloud cpcloud merged commit 1c7c6e3 into ibis-project:main Jun 3, 2024
78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cpcloud cpcloud cpcloud approved these changes

@jcrist jcrist Awaiting requested review from jcrist

Assignees
No one assigned
Labels
pyspark The Apache PySpark backend streaming Issue related to streaming APIs or backends
Projects
None yet
Milestone
9.1
Development

Successfully merging this pull request may close these issues.
2 participants
@chloeh13q @cpcloud