perf: prune parquet row groups when is_not_null is used

[taki-mekhalfa]

Filtering on non null values is more used in the wild than filtering on null values.

I changed the BatchStats to include the number of rows in that rg and compare it with the number of nulls statistic if available.

I also tried to simplify not(is_null) to is_not_null as I see this pattern a lot, this will allow to apply the pruning in these cases. [I also transformed not(is_not_null) to is_null]

>>> pl.scan_parquet('examples/datasets/null_nutriscore.parquet').collect()
shape: (27, 5)
┌──────────┬──────────┬────────┬──────────┬─────────────┐
│ category ┆ calories ┆ fats_g ┆ sugars_g ┆ nutri_score │
│ ---      ┆ ---      ┆ ---    ┆ ---      ┆ ---         │
│ str      ┆ i64      ┆ f64    ┆ i64      ┆ str         │
╞══════════╪══════════╪════════╪══════════╪═════════════╡
│ seafood  ┆ 117      ┆ 9.0    ┆ 0        ┆ null        │
│ seafood  ┆ 201      ┆ 6.0    ┆ 1        ┆ null        │
│ fruit    ┆ 59       ┆ 1.0    ┆ 14       ┆ null        │
│ meat     ┆ 97       ┆ 6.0    ┆ 0        ┆ null        │
│ meat     ┆ 124      ┆ 12.0   ┆ 1        ┆ null        │
│ …        ┆ …        ┆ …      ┆ …        ┆ …           │
│ seafood  ┆ 155      ┆ 5.0    ┆ 0        ┆ null        │
│ fruit    ┆ 133      ┆ 0.0    ┆ 27       ┆ null        │
│ seafood  ┆ 205      ┆ 9.0    ┆ 0        ┆ null        │
│ fruit    ┆ 72       ┆ 4.5    ┆ 7        ┆ null        │
│ fruit    ┆ 60       ┆ 1.0    ┆ 7        ┆ null        │
└──────────┴──────────┴────────┴──────────┴─────────────┘

>>> pl.scan_parquet('examples/datasets/null_nutriscore.parquet').filter(pl.col('nutri_score').is_not_null()).collect()
parquet file can be skipped, the statistics were sufficient to apply the predicate.
shape: (0, 5)
┌──────────┬──────────┬────────┬──────────┬─────────────┐
│ category ┆ calories ┆ fats_g ┆ sugars_g ┆ nutri_score │
│ ---      ┆ ---      ┆ ---    ┆ ---      ┆ ---         │
│ str      ┆ i64      ┆ f64    ┆ i64      ┆ str         │
╞══════════╪══════════╪════════╪══════════╪═════════════╡
└──────────┴──────────┴────────┴──────────┴─────────────┘

>>> pl.scan_parquet('examples/datasets/null_nutriscore.parquet').filter(~pl.col('nutri_score').is_null()).collect()
parquet file can be skipped, the statistics were sufficient to apply the predicate.
shape: (0, 5)
┌──────────┬──────────┬────────┬──────────┬─────────────┐
│ category ┆ calories ┆ fats_g ┆ sugars_g ┆ nutri_score │
│ ---      ┆ ---      ┆ ---    ┆ ---      ┆ ---         │
│ str      ┆ i64      ┆ f64    ┆ i64      ┆ str         │
╞══════════╪══════════╪════════╪══════════╪═════════════╡
└──────────┴──────────┴────────┴──────────┴─────────────┘

edit: typos and layout

[ritchie46]

Good addition. Thank you @taki-mekhalfa

[ritchie46]

Yeap, good one. 👍