fix(python): Use primitive constructors to create a Series of lists when dtype is provided

[petrosbar]

When a Series of lists is constructed, it uses the AnyValue interface even when the dtype is provided. Besides the performance implications, this often leads to incorrect results because even though the type is provided, it is inferred anyhow and then, if the inferred type differs from the provided type, the result is cast to the latter.

Here are some examples of incorrect cases:

import polars as pl

u64_max = (2**64) - 1

# Incorrect case (1)*

pl.Series("a", [[u64_max]], dtype=pl.List(pl.UInt64))
# OverflowError: Python int too large to convert to C long


# Incorrect case (2); similar to (1) but not the same

pl.Series("a", [[u64_max, 1]], dtype=pl.List(pl.UInt64))
# SchemaError: unexpected value while building Series of type Float64; found value of type UInt64: 18446744073709551615

*This case has been reported in #14277.

This PR changes the way a Series of lists is constructed when the dtype is provided, while still falling back to AnyValue when it is not. It does so by recursively applying sequence_to_pyseries to every inner sublist and eventually calling the primitive constructor, e.g., PySeries.new_opt_u64, that corresponds to the given dtype.

This results in:

1. Faster construction.
2. Fixing incorrect cases mentioned above.
3. Series of list construction inherits the same semantics (e.g., strictness) of the primitive constructors.

Note that I had to edit a couple of faulty test cases that emerged due to this change.

Partially closes #14277. The pl.Array case can be addressed in a separate PR.

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 80.98%. Comparing base (c5b53ab) to head (f0cf33e).

❗ Current head f0cf33e differs from pull request most recent head 5428252. Consider uploading reports for the commit 5428252 to get more accurate results

Files	Patch %	Lines
py-polars/polars/_utils/construction.py	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15002      +/-   ##
==========================================
- Coverage   80.98%   80.98%   -0.01%     
==========================================
  Files        1333     1333              
  Lines      173174   173176       +2     
  Branches     2458     2460       +2     
==========================================
- Hits       140252   140240      -12     
- Misses      32454    32468      +14     
  Partials      468      468              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stinodego]

The fix looks good, but why were some of the test cases changed? These shouldn't be affected, right?

If you could revert those changes, I'll merge this. (the newly added test cases can stay of course).

EDIT: Nevermind, the test changes are valid 👍

[stinodego]

I'm not super happy that we're doing a full loop over the values on the Python side, but the logic in the PyO3 AnyValue code for this is much worse. So this should definitely be an improvement.

[petrosbar]

The fix looks good, but why were some of the test cases changed? These shouldn't be affected, right?

If you could revert those changes, I'll merge this. (the newly added test cases can stay of course).

EDIT: Nevermind, the test changes are valid 👍

For clarity and future reference, one example of an edited test case is the following:

Previously, the following was used in a test case:

s = pl.Series(
    values=[[10e10, 10e15], [10e12, 10e13], [10e10, 10e15]],
    dtype=pl.List(pl.Int64),
)

With AnyValue this would silently cast to integers.

But a similar non-nested case like the following:

s = pl.Series(
    values=[10e10, 10e15],
    dtype=pl.Int64,
)

would raise with:
TypeError: 'float' object cannot be interpreted as an integer

Other edits are similar.

[petrosbar]

I'm not super happy that we're doing a full loop over the values on the Python side, but the logic in the PyO3 AnyValue code for this is much worse. So this should definitely be an improvement.

100% agree. I think this PR was a quick win but it looks like there's more room for improvement here. Thanks for the review!