from_pandas: pyo3_runtime.PanicException: cannot set validity of a null array

[gam-phon]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

I am trying to transfer between pandas and polars and found this issue when loading data from pandas to polars, if I load them directly it will be no issue but from_pandas cause the issue.

The element sometime is dict and another is None. that is when the issue exists.

Reproducible example

import pandas as pd
import polars as pl

data = [
    {
        "a": {
            "b": None,
        },
    },
    {"a": None},
]

# No issue
df_pandas = pd.DataFrame(data)
print(df_pandas)
# No issue
df_polars = pl.DataFrame(data)
print(df_polars)

# Here is the issue
pl.from_pandas(df_pandas)


data = [
    {
        "a": {
            "b": None,
        },
    },
    {"a": {}},
]
# No issue
pl.from_pandas(pd.DataFrame(data))

data = [
    {
        "a": {
            "b": 1,
        },
    },
    {"a": None},
]

# No issue
pl.from_pandas(pd.DataFrame(data))

data = [
    {
        "a": {
            "b": None,
        },
    },
]

# No issue
pl.from_pandas(pd.DataFrame(data))

Expected behavior

I know that the data maybe is corrupted but loading them to polars directly has no issue so I am expecting same behavior in both (pl.from_pandas and pl.DataFram) either raise error in both or ignore and handle in both

Installed versions

---Version info---
Polars: 0.15.16
Index type: UInt32
Platform: macOS-13.1-arm64-arm-64bit
Python: 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)]
---Optional dependencies---
pyarrow: 10.0.1
pandas: 1.5.3
numpy: 1.24.1
fsspec: <not installed>
connectorx: 0.3.2_alpha.1
xlsx2csv: <not installed>
deltalake: <not installed>
matplotlib: <not installed>
None

[gam-phon]

Maybe this is related to

• gracefully handle conversion of object dtype with nans #6107