Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Split replace functionality into two separate methods #16921

Merged
merged 5 commits into from
Jun 22, 2024

Conversation

stinodego
Copy link
Member

@stinodego stinodego commented Jun 13, 2024

Closes #14302

Changes

  • Add replace_strict method to Expr and Series. This method is similar to what replace was previously, with the exception that it raises an error if any non-null values were not replaced. Specify a default to set all values that were not replaced to that default.
  • replace now always keeps the existing data type - this is a breaking change (see example below). Thedefault and return_dtype parameters are deprecated.

Example

Before:

>>> s = pl.Series([1, 2, 3])
>>> s.replace(1, "a")
shape: (3,)
Series: '' [str]
[
        "a"
        "2"
        "3"
]

After:

>>> s.replace(1, "a")
Traceback (most recent call last):
...
polars.exceptions.InvalidOperationError: conversion from `str` to `i64` failed in column 'literal' for 1 out of 1 values: ["a"]
>>> s.replace_strict(1, "a", default=s)
shape: (3,)
Series: '' [str]
[
        "a"
        "2"
        "3"
]

@github-actions github-actions bot added breaking Change that breaks backwards compatibility enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Jun 13, 2024
Copy link

codspeed-hq bot commented Jun 13, 2024

CodSpeed Performance Report

Merging #16921 will not alter performance

Comparing replace-strict (2dc0d1f) with main (69b8440)

Summary

✅ 37 untouched benchmarks

@stinodego stinodego force-pushed the replace-strict branch 3 times, most recently from bf621f6 to a6bccde Compare June 18, 2024 13:49
Copy link

codecov bot commented Jun 18, 2024

Codecov Report

Attention: Patch coverage is 95.95960% with 8 lines in your changes missing coverage. Please review.

Project coverage is 80.88%. Comparing base (8a6bf4b) to head (2dc0d1f).

Files Patch % Lines
crates/polars-ops/src/series/ops/replace.rs 96.80% 4 Missing ⚠️
py-polars/src/lazyframe/visitor/expr_nodes.rs 0.00% 2 Missing ⚠️
crates/polars-plan/src/dsl/function_expr/mod.rs 80.00% 1 Missing ⚠️
crates/polars-plan/src/dsl/mod.rs 95.65% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16921      +/-   ##
==========================================
+ Coverage   80.86%   80.88%   +0.02%     
==========================================
  Files        1456     1456              
  Lines      191141   191297     +156     
  Branches     2728     2734       +6     
==========================================
+ Hits       154562   154727     +165     
+ Misses      36073    36064       -9     
  Partials      506      506              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@stinodego stinodego force-pushed the replace-strict branch 5 times, most recently from dadc13f to aeacbf8 Compare June 19, 2024 22:59
@stinodego stinodego marked this pull request as ready for review June 20, 2024 09:32
@stinodego stinodego added the deprecation Add a deprecation warning to outdated functionality label Jun 20, 2024
@stinodego stinodego marked this pull request as draft June 20, 2024 10:39
@stinodego stinodego marked this pull request as ready for review June 20, 2024 20:36
@stinodego
Copy link
Member Author

@ritchie46 This PR is now ready for review. Design is a bit different from what we discussed in the office:

  1. Replacing nulls by a different value is supported. Reasons I went for this:
  • We were already supporting it.
  • It would be weird if replace(1, None) works but replace(None, 1) doesn't.
  • It may be more efficient to replace everything in one go, rather than calling replace followed by fill_null.
  1. Nulls from the original Series do not propagate to the output. This really doesn't make sense, especially when the user species a full column as the default.

} else {
self.apply_many_private(FunctionExpr::Replace { return_dtype }, &args, false, false)
self.apply_many_private(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR, but I think we can set this to map later. I don't believe this is group-aware. If the input is group-aware, that would already bubble up that info.

let mut out = new.new_from_index(0, s.len());

// Transfer validity from `mask` to `out`.
if mask.null_count() > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a future PR we must optimize this. Adding a validity is almost free, but here it takes a very expensive branchy path.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I wasn't happy about this either - but I couldn't find a way to do this without a lot of code... I figure we're probably missing some API building blocks here.

Copy link
Member

@ritchie46 ritchie46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is still a few things left to optimize, but let's ensure we have the correct behavior for 1.0 first. 👍

@ritchie46 ritchie46 merged commit 0d3a285 into main Jun 22, 2024
28 checks passed
@ritchie46 ritchie46 deleted the replace-strict branch June 22, 2024 07:07
Wouittone pushed a commit to Wouittone/polars that referenced this pull request Jun 22, 2024
@stinodego stinodego changed the title feat!: Split replace functionality into two separate functions feat!: Split replace functionality into two separate methods Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Change that breaks backwards compatibility deprecation Add a deprecation warning to outdated functionality enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Split replace functionality into two separate functions
2 participants