some prefetch/gather benchmarks

[ctb]

A comparison of a Rust counter-gather implementation (pymagsearch) with sourmash gather from sourmash v4.5.0.

table hackmd for nice editing

command	wall time	max RSS	CPU time	thread efficiency
sourmash gather	9:07.73	2,661 MB	489.02 s	0.89
pymagsearch develop mode	5:17.82	113 MB	8735.05 s	27.6x
pymagsearch release mode	1:28.76	107 MB	298.79	3x

ht Rob Patro for suggesting release mode 😆

pymagsearch gather.py with 32 threads -

built with default develop flags.

% /usr/bin/time -v ./gather.py SRR606249.k31.sig.gz gtdb-rs207-k31.list.txt  --output-prefetch p.csv --output-gather g.csv
...
Command being timed: "./gather.py SRR606249.k31.sig.gz gtdb-rs207-k31.list.txt --output-prefetch p.csv --output-gather g.csv"
        User time (seconds): 8735.05
        System time (seconds): 15.75
        Percent of CPU this job got: 2753%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 5:17.82
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 113036
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 35
        Minor (reclaiming a frame) page faults: 310501
        Voluntary context switches: 142072
        Involuntary context switches: 961049
        Swaps: 0
        File system inputs: 8456
        File system outputs: 72
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

sourmash gather with 1 thread

% /usr/bin/time -v sourmash gather SRR606249.k31.sig.gz /group/ctbrowngrp/sourmash-db/gtdb-rs207/gtdb-rs207.genomic-reps.dna.k31.zip --save-prefetch-csv p2.csv -o g2.csv
...
        Command being timed: "sourmash gather SRR606249.k31.sig.gz /group/ctbrowngrp/sourmash-db/gtdb-rs207/gtdb-rs207.genomic-reps.dna.k31.zip --save-prefetch-csv p2.csv -o g2.csv"
        User time (seconds): 489.02
        System time (seconds): 4.30
        Percent of CPU this job got: 90%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 9:07.73
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2661284
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 7
        Minor (reclaiming a frame) page faults: 702574
        Voluntary context switches: 2058
        Involuntary context switches: 749858
        Swaps: 0
        File system inputs: 3547408
        File system outputs: 544
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

pymagsearch gather.py with 32 threads -

built with maturin develop --release using suggestions from Rob

removing GCF_014217355.1 Fusobacterium hwasookii strain=KCOM 1249, ASM1421735v1
        Command being timed: "./gather.py SRR606249.k31.sig.gz gtdb-rs207-k31.li
st.txt --output-prefetch p.csv --output-gather g.csv"
        User time (seconds): 298.79
        System time (seconds): 9.04
        Percent of CPU this job got: 346%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:28.76
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 107128
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 128932
        Voluntary context switches: 242522
        Involuntary context switches: 27046
        Swaps: 0
        File system inputs: 7146392
        File system outputs: 72
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[ctb]

note @nomad421 advice: https://twitter.com/nomad421/status/1582798621107314689

RUSTFLAGS="-C target-cpu=native" cargo build --release

[ctb]

note that the memory usage on the Python side may not be real. working on it here - sourmash-bio/sourmash#2340

[ctb]

Very out of date!