[WIP] Add "knn" and "umap" commands

[olgabot]

Depending on #925 working, then can create a k-nearest neighbor graph and umap visualization directly from a sourmash SBT index.

• Is it mergeable?
• make test Did it pass the tests?
• make coverage Is the new code covered?
• Did it change the command-line interface? Only additions are allowed
  without a major version increment. Changing file formats also requires a
  major version number increment.
• Was a spellchecker run on the source code and documentation after
  changes were made?

[luizirber]

Continuing this conversation:

olgabot

Is this because SBTs are dependent on the order in which they were constructed?

titus

yes, currently SBTs are order dependent.

olgabot

So there’s no guarantee that leaves sharing parents of nodes further down are more similar than leaves sharing higher-level parents?

titus

that is correct.
I think Luiz has been working on ways to build the SBT differently, as have others, but if you want to enable online building and node additions, you end up with a lot of balance challenges.

luizirber

Yup. But clustering similar nodes would also make searching (especially gather) way faster...
There is a scaffold command on the Rust side which will take a list of datasets and cluster them, but it is not online
I was thinking to do something like:

• index collects datasets up to a threshold (~100?), and just do a linear search til then
• when there are enough datasets, run scaffold to build a skeleton for new updates (see the Allsome SBT paper for a discussion about clustering too early)
• modify the add_node call to search for best position to put the new dataset. This can be the first thing to change, but without proper tree balancing it will mess the internal SBT representation...
  SBTs behave better if they are clustered by shared content, and if the tree is dense (don't increase depth if there are positions available)

[olgabot]

The current implementation of SBTs is not properly localized, so a nearest neighbor graph built on the SBTs and then fed into a UMAP implementation in this PR. Instead of getting the correct plot on the left that was built on an all-by-all matrix csv from sourmash compare, instead one gets the plot on the right which has mini-versions of the plot on the right.

This is due to the SBTs not being properly localized, as the nearest neighbor algorithm depends on leaves sharing parents to be more similar in jaccard space than leaves that do not share parents.

[olgabot]

Resurrecting this PR...

[sonarcloud]

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities (and 0 Security Hotspots to review)
13 Code Smells

No Coverage information
0.0% Duplication