fastmultigather should report full gather results for non rocksdb databases!
#287
Comments
4 tasks
bluegenes
added a commit
that referenced
this issue
May 10, 2024
…multigather` (#298) This PR adds utilities for building full gather results file for `fastgather` and non-rocksdb `fastmultigather`, and makes full output default. - Fixes #287 - Fixes #187 - Fixes #254 - includes a local fix for #318, which means that the `fastgather` and **non-rocksdb** `fastmultigather` full output here matches sourmash gather. Issues with rocksdb gather are being handled elsewhere. ## Benchmarking | software/version | command | details | time | max RAM | | -------- | -------- | -------- | -- | -- | | branchwater v0.9.3 | `fastgather` | minimal result | <span style="color:green">**1m 47s**</span> | <span style="color:black">**14 GB**</span> | | branchwater v0.9.3-dev | `fastgather` | full result | <span style="color:green">**1m 57s**</span> | <span style="color:black">**14 GB**</span> | | branchwater v0.9.3 | `fastmultigather` | minimal result | <span style="color:red">**8m 3s**</span> | <span style="color:black">**25 GB**</span> | | branchwater v0.9.3-dev | `fastmultigather` | full result | <span style="color:red">**8m 9s**</span> | <span style="color:black">**25 GB**</span> | | branchwater v0.9.3 | `fastmultigather` | rocksdb full result | <span style="color:green">**24s**</span> | <span style="color:green">**600 MB**</span> | progress/separate PRs: - [x] Fill out `match_filename` in full results (#303; requires new sourmash core release with sourmash-bio/sourmash#3121) - [x] switch to using `KmerMinHashBTree` for hash subtraction +benchmark. Per luiz, `KmerMinHashBTree` are better for any situation where we'll be subtracting/adding hashes to a sketch #310 - [x] sourmash: make getting `Record`.filename public in order to keep match_filename and write it to full results. (sourmash-bio/sourmash#3121) - [x] remove --full-results and make full results default #327 --------- Co-authored-by: C. Titus Brown <titus@idyll.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ref sourmash-bio/sourmash#3095
I don't think there's any real impediment, just needs to happen! :)
The text was updated successfully, but these errors were encountered: