Leverage the power of polars across qsv commands

[13minutes-yt]

Is it possible to implement qsv count taxi.csv to search if polars feature is installed and utilize polars instead.
qsv sqlp -Q taxi.csv "select count(*) from read_csv('taxi.csv'). it will make it much faster.

[ondohotola]

What prevents you from using that sqlp query?

[13minutes-yt]

I am Lazy . Actually I use a bash alias wc1 for that., but it applies for filenames not stdin

[ondohotola]

Great motivation for an Open Source project…

[jqnatividad]

Is it possible to implement qsv count taxi.csv to search if polars feature is installed and utilize polars instead. qsv sqlp -Q taxi.csv "select count(*) from read_csv('taxi.csv'). it will make it much faster.

This is a great idea @13minutes-yt !

Some quick timings :

# unindexed taxi.csv
/usr/bin/time qsv count taxi.csv
7433139
        0.89 real         0.81 user         0.05 sys

# sqlp count is 9x faster!
/usr/bin/time qsv sqlp taxi.csv "select count(*) from _t_1"
len
7433128
(1, 1)
        0.10 real         0.61 user         0.09 sys

# create an index
/usr/bin/time qsv index taxi.csv                      
        0.89 real         0.81 user         0.06 sys

# indexed count is still faster
/usr/bin/time qsv count taxi.csv                           
7433139
        0.01 real         0.00 user         0.00 sys

So if the polars feature is enabled, I'm thinking of adding logic in the count command to use sqlp if the CSV file has no index.

Curiously though, sqlp returns a different count... I'll have to dig into that and report it to the Polars project if its a bug...

[13minutes-yt]

it seems the different count is due to csv parsing. I ran polars cli with default output (table) . It gives correct count while on csv (polars -o csv) it gave a discrepancy.

[jqnatividad]

@13minutes-yt , just merged #1656

Will find other opportunities to sprinkle Polars goodness across qsv 😄 (next up - pivot)

[13minutes-yt]

The under counting bug has been fixed with polars-rust 0.38.2
pola-rs/polars#14963

[jqnatividad]

Yep. Saw that too... upgraded to 0.38.2 on master.

Aiming to release qsv 0.124 this week...