Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect writing of NULL rows for struct column containing Decimal with precision >18 to Parquet #19448

Closed
2 tasks done
nameexhaustion opened this issue Oct 25, 2024 · 0 comments · Fixed by #19476
Closed
2 tasks done
Assignees
Labels
A-dtype-decimal Area: decimal data type A-io-parquet Area: reading/writing Parquet files accepted Ready for implementation bug Something isn't working P-low Priority: low python Related to Python Polars

Comments

@nameexhaustion
Copy link
Collaborator

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import io
import polars as pl

pl.DataFrame(
    {"x": [{"a": None}, None]}, schema={"x": pl.Struct({"a": pl.Decimal(19, 2)})}
).write_parquet(f := io.BytesIO())
f.seek(0)
print(pl.read_parquet(f))

# Output
# shape: (2, 1)
# ┌───────────┐
# │ x         │
# │ ---       │
# │ struct[1] │
# ╞═══════════╡
# │ null      │
# │ null      │
# └───────────┘
#
# Expected
# shape: (2, 1)
# ┌───────────┐
# │ x         │
# │ ---       │
# │ struct[1] │
# ╞═══════════╡
# │ {null}    │
# │ null      │
# └───────────┘

Log output

No response

Issue description

The outer validity gets incorrectly set.

Expected behavior

See example code

Installed versions

1.11.0

@nameexhaustion nameexhaustion added bug Something isn't working python Related to Python Polars needs triage Awaiting prioritization by a maintainer A-dtype-decimal Area: decimal data type P-medium Priority: medium A-io-parquet Area: reading/writing Parquet files and removed needs triage Awaiting prioritization by a maintainer labels Oct 25, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Oct 25, 2024
@nameexhaustion nameexhaustion added P-low Priority: low and removed P-medium Priority: medium labels Oct 25, 2024
@nameexhaustion nameexhaustion added the accepted Ready for implementation label Oct 25, 2024
@coastalwhite coastalwhite self-assigned this Oct 27, 2024
coastalwhite added a commit to coastalwhite/polars that referenced this issue Oct 27, 2024
This implements writing the writing of arbitrary nested `FixedSizeBinary`,
which in particular allows for the writing of nested high-precision decimals.
This removes the silent data-loss that was happening before.

Fixes pola-rs#19448.
@github-project-automation github-project-automation bot moved this from Ready to Done in Backlog Oct 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-dtype-decimal Area: decimal data type A-io-parquet Area: reading/writing Parquet files accepted Ready for implementation bug Something isn't working P-low Priority: low python Related to Python Polars
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants