Polars' rust parquet engine reads/writes files that are unreadable by duckdb/pandas/pyarrow (use_pyarrow=True)

[theelderbeever]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

Write parquet using Rust Arrow

import polars as pl
import pandas as pd
import duckdb

pl.DataFrame([
    {"struct": {"field": None}}
]).write_parquet("null.parquet")

pl.read_parquet("null.parquet") # Successful
pl.read_parquet("null.parquet", use_pyarrow=True) # Fails
pd.read_parquet("null.parquet") # Fails
duckdb.sql("select * from 'null.parquet'") # Fails

Write parquet using pyarrow

import polars as pl
import pandas as pd
import duckdb

pl.DataFrame([
    {"struct": {"field": None}}
]).write_parquet("null.parquet", use_pyarrow=True)

pl.read_parquet("null.parquet") # Successful
pl.read_parquet("null.parquet", use_pyarrow=True) # Successful
pd.read_parquet("null.parquet") # Successful
duckdb.sql("select * from 'null.parquet'") # Successful

Log output

No response

Issue description

Polars' Rust parquet writer writes Dataframes containing a null type struct field Schema([('struct', Struct({'field': Null}))]) in a format unreadable by other parquet readers (pyarrow/duckdb). However, using use_pyarrow=True writes a file readable by the other readers.

Expected behavior

Parquet files written using any engine should be compatible with one another.

Installed versions

df.select(df.select("recurring").unnest("recurring").to_struct()).write_parquet(
    "recurring_unnested.parquet"
)

[ritchie46]

@coastalwhite can you take a look?

[coastalwhite]

It seems a problem with the conversions to Arrow schema's. Adding a schema seems to fix the problem for now.

import polars as pl
import duckdb

pl.DataFrame(
    [ {"struct": {"field": None}} ],
    schema = { 'struct': pl.Struct({ 'field': pl.Int64 }) }
).write_parquet("null.parquet")

print(duckdb.sql("select * from 'null.parquet'")) # Successful

Will investigate what we are doing wrong.