Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Factor out logic for re-use by new streaming CSV source #19637

Merged
merged 4 commits into from
Nov 6, 2024

Conversation

nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Nov 5, 2024

Some prep work.

  • Adds support parsing Time when using scan_csv - currently it only worked with read_csv
  • Adds a POLARS_FORCE_NEW_STREAMING flag

@github-actions github-actions bot added internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars labels Nov 5, 2024
Copy link

codecov bot commented Nov 5, 2024

Codecov Report

Attention: Patch coverage is 94.21965% with 10 lines in your changes missing coverage. Please review.

Project coverage is 79.91%. Comparing base (0f64785) to head (9711fbf).
Report is 42 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-io/src/csv/read/reader.rs 88.63% 5 Missing ⚠️
crates/polars-lazy/src/frame/mod.rs 66.66% 2 Missing ⚠️
...rates/polars-time/src/chunkedarray/string/infer.rs 81.81% 2 Missing ⚠️
crates/polars-io/src/csv/read/schema_inference.rs 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #19637      +/-   ##
==========================================
- Coverage   79.92%   79.91%   -0.02%     
==========================================
  Files        1536     1536              
  Lines      211686   211670      -16     
  Branches     2445     2445              
==========================================
- Hits       169192   169156      -36     
- Misses      41939    41959      +20     
  Partials      555      555              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46
Copy link
Member

As a side effect this also adds support parsing Time when reading CSV.

How come?

No logical changes further? E.g. no perf changes.

@ritchie46 ritchie46 merged commit 750b371 into pola-rs:main Nov 6, 2024
26 checks passed
tylerriccio33 pushed a commit to tylerriccio33/polars that referenced this pull request Nov 8, 2024
@c-peters c-peters added the accepted Ready for implementation label Nov 11, 2024
@nameexhaustion nameexhaustion deleted the csv-src branch November 18, 2024 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants