feat: Decompress in CSV / NDJSON scan

[nameexhaustion]

We can have support for this without too much trouble if we concede that we end up fully decompressing twice (once during IR conversion / file_info, and then again during the actual read) - I can open an issue afterwards for that.

print(
    pl.read_ndjson(
        "hf://datasets/Anthropic/hh-rlhf/helpful-rejection-sampled/train.jsonl.gz"
    )
)

Before

polars.exceptions.ComputeError: stream did not contain valid UTF-8 at line 0

After

shape: (52_421, 2)
┌───────────────────────────────┬───────────────────────────────┐
│ chosen                        ┆ rejected                      │
│ ---                           ┆ ---                           │
│ str                           ┆ str                           │
╞═══════════════════════════════╪═══════════════════════════════╡
│                               ┆                               │
│                               ┆                               │
│ Human: If I was arrested for… ┆ Human: If I was arrested for… │
│                               ┆                               │
│                               ┆                               │
│ Human: What tools do I need … ┆ Human: What tools do I need … │
│                               ┆                               │
│                               ┆                               │
│ Human: What are some good ex… ┆ Human: What are some good ex… │
│                               ┆                               │
│                               ┆                               │
│ Human: I'm moving to Utah ne… ┆ Human: I'm moving to Utah ne… │
│                               ┆                               │
│                               ┆                               │
│ Human: What is an easy to ma… ┆ Human: What is an easy to ma… │
│ …                             ┆ …                             │
│                               ┆                               │
│                               ┆                               │
│ Human: How do I teach a chil… ┆ Human: How do I teach a chil… │
│                               ┆                               │
│                               ┆                               │
│ Human: Is there life on othe… ┆ Human: Is there life on othe… │
│                               ┆                               │
│                               ┆                               │
│ Human: What is the proper wa… ┆ Human: What is the proper wa… │
│                               ┆                               │
│                               ┆                               │
│ Human: What does an ultrasou… ┆ Human: What does an ultrasou… │
│                               ┆                               │
│                               ┆                               │
│ Human: Are there any risks o… ┆ Human: Are there any risks o… │
└───────────────────────────────┴───────────────────────────────┘

Closes #7287

[codecov]

Codecov Report

Attention: Patch coverage is 84.44444% with 14 lines in your changes missing coverage. Please review.

Project coverage is 80.49%. Comparing base (0c2cfae) to head (6dcb65b).
Report is 3 commits behind head on main.

Files	Patch %	Lines
crates/polars-io/src/utils/other.rs	69.56%	7 Missing ⚠️
crates/polars-mem-engine/src/executors/scan/csv.rs	88.88%	2 Missing ⚠️
...tes/polars-mem-engine/src/executors/scan/ndjson.rs	88.88%	2 Missing ⚠️
crates/polars-utils/src/io.rs	50.00%	2 Missing ⚠️
...ipe/src/executors/sinks/group_by/generic/source.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17841      +/-   ##
==========================================
- Coverage   80.50%   80.49%   -0.02%     
==========================================
  Files        1504     1505       +1     
  Lines      197152   197202      +50     
  Branches     2805     2806       +1     
==========================================
+ Hits       158722   158737      +15     
- Misses      37909    37944      +35     
  Partials      521      521              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

We can have support for this without too much trouble if we concede that we end up fully decompressing twice (once during IR conversion / file_info, and then again during the actual read) - I can open an issue afterwards for that.

Can't we decompress into our file cache?

[nameexhaustion]

Can't we decompress into our file cache?

It's possible, but I will need to think a bit about how we can do it.

[ritchie46]

Yes, on second thought. Let's give it some thought. Eventually we'd like to stream decompressions. In bio-informatics they compress 100sGB's of csv.