perf(python): Improve Series.to_numpy performance for chunked Series that would otherwise be zero-copy

[stinodego]

Ref #16267

Instead of iterating over the values, we rechunk and create a writable view.

This regressed in #16178 - now we get the best of both worlds with a writable array and only a single, fast copy.

Performance of converting a chunked Series of 50 million float32s:

• Before: ~60ms
• After: ~25ms

I added some benchmark tests so that we may catch regressions here in the future.

[codecov]

Codecov Report

Attention: Patch coverage is 98.24561% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 80.75%. Comparing base (6d48c11) to head (319d255).
Report is 8 commits behind head on main.

Files	Patch %	Lines
py-polars/src/to_numpy.rs	98.14%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16301      +/-   ##
==========================================
- Coverage   80.78%   80.75%   -0.03%     
==========================================
  Files        1393     1393              
  Lines      179362   179455      +93     
  Branches     2921     2922       +1     
==========================================
+ Hits       144894   144919      +25     
- Misses      33965    34033      +68     
  Partials      503      503              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

Nice. I like this approach. :)

I added some benchmark tests so that we may catch regressions here in the future.

Not entirely sure how the codspeed works, but are these tests ran by codspeed?

[stinodego]

Not entirely sure how the codspeed works, but are these tests ran by codspeed?

Yes! You can see it here, 3 new benchmarks:
https://codspeed.io/pola-rs/polars/branches/to-np-copy-chunk

Also, I spotted a problem with this implementation for nested data, going to have another look before putting it in review again.

[ritchie46]

How does one register them then? A specific folder?

[stinodego]

How does one register them then? A specific folder?

You just have to do @pytest.mark.benchmark. That's all. I made a dedicated benchmark folder because our tests require some data generation utilities and it's nice to have those in one place, but it doesn't have to be in there necessarily.

[stinodego]

I had to add an up-front check whether the Series has the right dtype / nested nulls, otherwise we could rechunk unnecessarily.

Should be good to go now, waiting for CI to turn green 🤞

[rhshadrach-8451]

I'm seeing this raise in 0.20.27; it did not raise for me in 0.20.26. Is this expected?

pl.concat(
    [
        pl.DataFrame({"a": [1, 1, 2], "b": [2, 3, 4]}),
        pl.DataFrame({"a": [1, 1, 2], "b": [2, 3, 4]}),
    ]
).to_numpy()
# PanicException: source slice length (3) does not match destination slice length (6)

I haven't confirmed it to be from this PR, but looked likely.

[rhshadrach-8451]

Apologies - jumped the gun here. #16288 looks more likely.