Commits on Jul 10, 2024

1. perf: batch nested Parquet decoding …

   This PR is a follow up to pola-rs#17462. This batches the collects in the nested
   Parquet decoders, with that we can also simplify the code quite a lot.
   
   I did a benchmark where we had one column `{ 'x': pl.List(pl.Int8) }` of length
   `10_000_000`. Then, we read that Parquet file 50 times. Here are the results.
   
   ```
   Benchmark 1: After Optimization
     Time (mean ± σ):      3.398 s ±  0.064 s    [User: 49.412 s, System: 4.362 s]
     Range (min … max):    3.311 s …  3.490 s    10 runs
   
   Benchmark 2: Before Optimization
     Time (mean ± σ):      4.135 s ±  0.015 s    [User: 59.506 s, System: 5.234 s]
     Range (min … max):    4.105 s …  4.149 s    10 runs
   
   Summary
     After Optimization ran
       1.22 ± 0.02 times faster than Before Optimization
   ```

   

   coastalwhite committed Jul 10, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 6e096aa
   • Browse repository at this point

   Copy the full SHA

   6e096aa View commit details

   Browse the repository at this point in the history