Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rust, python): add streamable udfs #6614

Merged
merged 1 commit into from
Feb 1, 2023
Merged

Conversation

ritchie46
Copy link
Member

@ritchie46 ritchie46 commented Feb 1, 2023

Any function that can operate in batches can now be used in the streaming engine.

(pl.DataFrame({
    "a": [1, 2, 3]
}).lazy()
.map(
    f = lambda df: df.with_columns(pl.col("a").alias("b")),
    schema={"a": pl.Int64, "b": pl.Int64},
    streamable=True,
)).collect(streaming=True)

closes #6603

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Feb 1, 2023
@ritchie46 ritchie46 merged commit dea9bbd into master Feb 1, 2023
Vincenthays pushed a commit to Vincenthays/polars that referenced this pull request Feb 9, 2023
@stinodego stinodego deleted the streamable_udfs branch August 7, 2023 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Issue while using sink_parquet on a LazyFrame for local file
2 participants