sourmash prefetch is ~10 times slower than sourmash gather?!

[ctb]

following the work done on #1771 with some speed fixes merged in #2123, I decided to run sourmash prefetch with the same query/database:

sudo time py-spy record -o latest.svg -- sourmash prefetch SRR10988543.10k.zip bins.10k.zip >& prefetch.out

and it took 600 seconds!

Not 100% sure what's going on here but it might be related to some of the ANI work that @bluegenes has been doing.

Flamegraph:

[bluegenes]

hmmm -- I was a bit worried about add'l calculations when adding ANI stuff.

Now that we're outputting all the information we need for ANI estimation (ksize, scaled, n_dataset_hashes, etc) in the prefetch csv file, we could actually skip ANI estimation entirely during prefetch

We could:

1. only report ANI for gather
2. separately annotate a prefetch or gather csv file with ANI results, if desired (akin to running sourmash tax annotate on a gather file, this would just add ANI estimates).

[ctb]

sure - this is just a marker so we remember to investigate more.

[bluegenes]

👍🏻!

I think for most normal use cases (prefetch --> gather), folks might not want ANI results/ might only care about ANI to gather results. So the speedup might be worth removing ANI from prefetch (unless someone can just make it faster, ofc). I implemented it this way bc I've been using multiple prefetch calls to get all-by-all comparisons, and originally did not have sufficient info in the prefetch csv.

Removing ANI from prefetch is straightforward (for me/you for future ref):

• remove this line: https://github.com/sourmash-bio/sourmash/blob/latest/src/sourmash/search.py#L376
• remove ['query_containment_ani', 'match_containment_ani', 'average_containment_ani', 'max_containment_ani', 'potential_false_negative'] from prefetch_cols, so we're not writing ANI columns (https://github.com/sourmash-bio/sourmash/blob/latest/src/sourmash/search.py#L354-L356)
• add a command to estimate ANI from a prefetch or gather csv using distance_utils.py containment_to_distance. I have this implemented as a standalone script here: https://github.com/bluegenes/2022-sgc-ani/blob/main/recalc-ani.py

[ctb]

#2132 seems to fix this, or at least speed it up by a lot...