feat: Change API for writing partitioned Parquet to reduce code duplication

[nameexhaustion]

Move the partition parameters to existing scan_* functions instead of creating an entirely new function per scan type. There is also a bit of pre-work for getting IPC support in.

Before

df.write_parquet_partitioned(root, ["a", "b"])

After

df.write_parquet("docs/data/hive_write/", partition_by=["a", "b"])

[nameexhaustion]

drive-by - we can enable parallel gather now that we are processing in chunks.

[codecov]

Codecov Report

Attention: Patch coverage is 86.04651% with 6 lines in your changes missing coverage. Please review.

Project coverage is 80.47%. Comparing base (64b45a8) to head (e0123c3).
Report is 8 commits behind head on main.

Files	Patch %	Lines
crates/polars-io/src/ipc/write.rs	0.00%	3 Missing ⚠️
crates/polars-io/src/partition.rs	72.72%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17586      +/-   ##
==========================================
+ Coverage   80.45%   80.47%   +0.02%     
==========================================
  Files        1484     1484              
  Lines      195341   195485     +144     
  Branches     2778     2776       -2     
==========================================
+ Hits       157154   157322     +168     
+ Misses      37675    37652      -23     
+ Partials      512      511       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Julian-J-S]

Very nice to consolidate those 2 functions!
My 2 cents on the api:

• Using pl.PartitionedWriteOptions(...) in a function is very "rusty" but imo also very "un-pythonic" and does not fit with any other polars function. Usually python api tries to solver everything using basic types like dicts/lists and avoid additional classes
• imo a partition_cols (arrow, pandas) / partition_by (polars) / partition (my fav.) as list[str] | None would be a much cleaner and simples api for python users

[etiennebacher]

* imo a `partition_cols` (arrow, pandas) / `partition_by` (polars) / `partition` (my fav.) as `list[str] | None` would be a much cleaner and simples api for python users

I also think this would be more readable. Maybe even a hive_partitioning argument for the symmetry with the read/scan functions?

[ritchie46]

The reason we are considering the Options route is because you will have mutual exclusive arguments. E.g. if you pass a bytes object, which is a valid argument for writing a single parquet file, doesn't make sense if you write a partitioned dataset. With these options you ensure that those combinations don't occur.

[Julian-J-S]

yeah, totally get that!

I think python and rust have very different approaches here

• python
  • easy, concise, short
  • raise on bad input
  • join for exmaple also has left_on, right_on and on which are exclusive
• rust
  • verbose, strict, umambiguous
  • prevent bad input by requirering "perfect" input

[stinodego]

We can just raise when path is a BytesIO and the other parameters indicate partitioning. We do that often enough in other places in the API.

I'd say add a TypedDict parameter partitioning is most sensible.

[ritchie46]

I'd say add a TypedDict parameter partitioning is most sensible.

If we go for mutual excl params (which I am willing to go when properly raised), I do really think we shouldn't have nested parameters. That's just worst of both options, as you still have the worse/"unpythonic" ergonomics, but no type guarantees.

@nameexhaustion can you create the needed parameters and mark them as unstable? Then we can feel out the waters.

[nameexhaustion]

Updated