parquet file saved with pyarrow 14.0.1 reports different column lengths #14902
Labels
A-interop-arrow
Area: interoperability with other Arrow implementations (such as pyarrow)
A-io-parquet
Area: reading/writing Parquet files
accepted
Ready for implementation
bug
Something isn't working
needs triage
Awaiting prioritization by a maintainer
P-medium
Priority: medium
python
Related to Python Polars
Checks
Reproducible example
I'm including the actual file here since that's the only way to troubleshoot the issue.
see full back trace below
pyarrow can open the file without issue
Additionally, this works
Log output
This is the RUST_BACKTRACE=full output
Issue description
polars views the column as only having 24 entries which seems to be related to there being 24 hours in a day (maybe).
Here's the metadata on the column
When I resave the file with pyarrow 15 (which polars doesn't have a problem with) then this is the metadata
so it seems that when it's saved without PLAIN encodings that it has a problem.
Expected behavior
It should recognize the column properly
Installed versions
The text was updated successfully, but these errors were encountered: