Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix decompress_impl for csv with n_rows set #17118

Merged
merged 6 commits into from
Jun 22, 2024

Conversation

Mottl
Copy link
Contributor

@Mottl Mottl commented Jun 21, 2024

This commit truncates the out buffer in decompress_impl to exactly n_rows (if set).

Returning all the out buffer from decompress_impl yields to a corrupted last row and, as a result, to further parse errors like this:

ComputeError(ErrString("could not parse `-` as dtype `f64` at column 'b' (column number 2)...

@github-actions github-actions bot added the fix Bug fix label Jun 21, 2024
#[test]
#[cfg(any(feature = "decompress", feature = "decompress-fast"))]
fn test_read_compressed() {
const COMPRESSED_CSV: &str = "../../examples/datasets/compressed.csv.gz";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create the compressed file on the fly (in-memory). That saves adding a file to the repo.

Copy link
Contributor Author

@Mottl Mottl Jun 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current compressed.csv.gz is randomly generated and the n_rows which yields to ComputeError is found manually. I'd leave compressed.csv.gz as-is or remove test_read_compressed completely, since it's mostly here for illustrating the initial issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can create a in memory random file that illustrates this. Otherwise removing the test and file is also fine. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test_read_compressed and a correspondent csv.gz file are removed 👍

Copy link

codecov bot commented Jun 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.88%. Comparing base (8a6bf4b) to head (6fb4b51).
Report is 15 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17118      +/-   ##
==========================================
+ Coverage   80.86%   80.88%   +0.02%     
==========================================
  Files        1456     1456              
  Lines      191141   191340     +199     
  Branches     2728     2739      +11     
==========================================
+ Hits       154562   154769     +207     
+ Misses      36073    36064       -9     
- Partials      506      507       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46 ritchie46 merged commit 3af8115 into pola-rs:main Jun 22, 2024
21 checks passed
@ritchie46 ritchie46 changed the title fixes decompress_impl for csv with n_rows set fix: Fix decompress_impl for csv with n_rows set Jun 22, 2024
@ritchie46
Copy link
Member

Thanks @Mottl

@github-actions github-actions bot added python Related to Python Polars rust Related to Rust Polars labels Jun 22, 2024
Wouittone pushed a commit to Wouittone/polars that referenced this pull request Jun 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants