Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Properly write nested NullArray in Parquet #17807

Merged

Conversation

coastalwhite
Copy link
Collaborator

Fixes #17795.

@coastalwhite coastalwhite changed the title Fix parquet nested null array write 17795 fix: Parquet nested null array write Jul 23, 2024
@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Jul 23, 2024
@coastalwhite coastalwhite changed the title fix: Parquet nested null array write fix: Properly write nested NullArray in Parquet Jul 23, 2024
Copy link

codecov bot commented Jul 23, 2024

Codecov Report

Attention: Patch coverage is 90.47619% with 10 lines in your changes missing coverage. Please review.

Project coverage is 80.49%. Comparing base (66f0026) to head (cbde319).
Report is 18 commits behind head on main.

Files Patch % Lines
crates/polars-arrow/src/bitmap/immutable.rs 86.53% 7 Missing ⚠️
crates/polars-utils/src/foreign_vec.rs 91.89% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17807      +/-   ##
==========================================
+ Coverage   80.47%   80.49%   +0.01%     
==========================================
  Files        1503     1504       +1     
  Lines      197115   197152      +37     
  Branches     2794     2805      +11     
==========================================
+ Hits       158628   158696      +68     
+ Misses      37973    37935      -38     
- Partials      514      521       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coastalwhite coastalwhite force-pushed the fix-parquet-nested-null-array-write-17795 branch from b4899ad to dd022a5 Compare July 23, 2024 15:14
@coastalwhite
Copy link
Collaborator Author

There is something going wrong here in the propagate_nulls resulting from the struct refactor. @ritchie46 could you have a look at this?

@ritchie46 ritchie46 merged commit 0c2cfae into pola-rs:main Jul 24, 2024
26 checks passed
@coastalwhite coastalwhite deleted the fix-parquet-nested-null-array-write-17795 branch July 24, 2024 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Polars' rust parquet engine reads/writes files that are unreadable by duckdb/pandas/pyarrow (use_pyarrow=True)
2 participants