MRG: Calculate all gather stats in rust; use for rocksdb gather

[bluegenes]

This PR adds a calculate_gather_stats function (and criterion benchmark) to calculate all gather statistics directly in rust. It makes use of this function in disk_revindex.rs, meaning that gather on a rocksdb database will now produce the full GatherResult.

Note that this PR does not change the python layer to use these stats. This is just a step on the way to that :).

New functions:

• Sketch::MinHash::sum_abunds() - sum abundances in a MinHash
• Sketch::MinHash::n_unique_kmers() - len(self) * self.scaled.
• Sketch::inflate(abunds_from) - inflate abundances in self using abund values from abunds_from MinHash
• Sketch::inflated_abundances(abunds_from) - same process as inflate, but just return the abundance vector and sum of abundances.
• ani_utils::ani_from_containment - a streamlined function to get ANI from containment. I suspect a lot of the time added for ANI calculation (ref: testing over in branchwater) is the way we calculate it in python (including calculating a total number of k-mers). This excludes anything not really needed for basic ANI without confidence intervals.
• ani_utils::ani_from_containment_ci - get high and low CI ANI from containment
• index::calculate_gather_stats - calculate all gather statistics that can be separated from the gather iteration and return the full GatherResult

The "expensive" calculations:

• abundance-weighted values require re-calculating intersections and/or manipulating the abundance vector

  Thankfully I think the new inflate/inflated_abundances code is pretty efficient? It uses the itertools merge_join_by strategy from https://github.com/sourmash-bio/sourmash/compare/lirber/itertools_merge. Thanks @luizirber :)

• abundance median and stddev currently require cloning abunds, not sure how big of an issue that is.
• ANI with confidence intervals

Punted Issues

• calculate all stats for rust linear gather #3020: calculate all gather stats for all other database types. start with linear.rs since it uses GatherResult struct already.
• rust: calculate gather stats in parallel #3021 try moving calculate_gather_stats outside of the gather functions to enable postprocessing in parallel.
• FFI for new rust functions #3022 : write ffi for the new functions?
  • sketch::minhash::sum_abunds
  • sketch::minhash::n_unique_kmers,
  • sketch::minhash::inflate
  • sketch::minhash::inflated_abundances
  • ani_utils::ani_from_containment
  • ani_utils::ani_from_containment_ci
  • ani_utils::prob_nothing_in_common

ref sourmash-bio/sourmash_plugin_branchwater#187.

[codecov]

Codecov Report

Attention: 18 lines in your changes are missing coverage. Please review.

Comparison is base (fa4ae0b) 86.71% compared to head (7c0853c) 86.74%.

Files	Patch %	Lines
src/core/src/ani_utils.rs	82.35%	9 Missing ⚠️
src/core/src/sketch/minhash.rs	87.80%	5 Missing ⚠️
src/core/src/index/mod.rs	85.71%	3 Missing ⚠️
src/core/src/ffi/minhash.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #2943      +/-   ##
==========================================
+ Coverage   86.71%   86.74%   +0.03%     
==========================================
  Files         135      136       +1     
  Lines       15396    15509     +113     
  Branches     2624     2624              
==========================================
+ Hits        13350    13453     +103     
- Misses       1745     1755      +10     
  Partials      301      301              

Flag	Coverage Δ
hypothesis-py	25.56% <ø> (ø)
python	92.85% <ø> (ø)
rust	61.06% <87.14%> (+1.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bluegenes]

There are a number of tests with missing coverage, is that intentional or a bug in coverage calculations?

Lost track of a few fns that needed testing. Added remaining tests.

There are a few codecov lines that I don't fully understand -- hoping they get updated as covered.

[luizirber]

This is so cool, thanks Tessa!

[luizirber]

I piggybacked some commits replacing Signature with SigStore in GatherResults, just to find out that the .unload() method on SigStore is not implemented yet, oops! But now things are in place for it to work =]

I also changed .remove_many() on KmerMinHash to take an Iterator, this way we don't need to allocate a new vec to remove hashes...

The last commit is for some clippy fixes, I suspect CI is not running clippy with all features enabled, and some got away