sum crc32

[tooptoop4]

help with #2121 (comment)

[jqnatividad]

Hi @tooptoop4 ,

Can you run qsv --version and share it here?

What kind of machine are you using? How big is the file? How many rows? How many columns? Can you share the output of qsv stats --everything as well? Did you index the file? How long did it take to run?

[tooptoop4]

qsv apply operations crc32 "Unique Key" --new-column myfile.csv -o /tmp/output.csv along is taking 76seconds for my 6GB file with 203 columns (comma delimiter with each col enclosed in quotes)

[jqnatividad]

And how long did your python script take? If you're not willing to share your script, is the script above materially equivalent?

How many rows is your 6gb, 203 column file? Originally, you said you had 800k rows and 150 columns, so this must be another file.

The throughput in my example is 419,463.08 rows/seconds (1,000,000 rows/2.384 seconds). And that is because qsv apply is multi-threaded and the crc32fast library is SIMD-accelerated if your platform is supported.

In comparison, the throughput for my binascii-crc32.py is 74,560.10 rows/second (1,000,000 rows/13.412 seconds).

I suspect your docker configuration is the constraining factor that's preventing qsv from running efficiently.

Have you tried running qsv apply operations crc32 outside your docker container?

Can you give the exact result of qsv --version inside your docker container and outside your docker container?

It will give you the number of threads qsv detected, and the memory available to it, which I'll require to help you troubleshoot your issue.

If you don't want to share diagnostic info I've been asking for since the beginning, then I'm afraid I can't help you any further.

[tooptoop4]

will be able to dig it up on monday, it took around 32 seconds in python, there was no pandas, it was a one liner along the lines of qsv select col numbers.txt | tail -n +2 | python -c "import sys, binascii; print(sum(binascii.crc32(line.encode()) for line in sys.stdin))"

[jqnatividad]

FWIW, with qsv 0.137.0, I get these figures:

$  hyperfine 'qsv apply operations crc32 "Unique Key" --new-column bobb NYC_311_SR_2010-2020-sample-1M.csv |  qsv sqlp - --cache-schema  "select sum(bobb) from _t_1"'
Benchmark 1: qsv apply operations crc32 "Unique Key" --new-column bobb NYC_311_SR_2010-2020-sample-1M.csv |  qsv sqlp - --cache-schema  "select sum(bobb) from _t_1"
  Time (mean ± σ):      2.802 s ±  0.008 s    [User: 3.076 s, System: 0.703 s]
  Range (min … max):    2.787 s …  2.813 s    10 runs

For your one-liner:

$ hyperfine 'qsv select 1 NYC_311_SR_2010-2020-sample-1M.csv | tail -n +2 | python -c "import sys, binascii; print(sum(binascii.crc32(line.encode()) for line in sys.stdin))"'
Benchmark 1: qsv select 1 NYC_311_SR_2010-2020-sample-1M.csv | tail -n +2 | python -c "import sys, binascii; print(sum(binascii.crc32(line.encode()) for line in sys.stdin))"
  Time (mean ± σ):     775.7 ms ±   3.3 ms    [User: 896.5 ms, System: 59.2 ms]
  Range (min … max):   769.6 ms … 779.3 ms    10 runs

So I recommend you stay with binascii, and just use qsv for selecting the column.