Add ability to adjust --num-hashes, --scaled on-the-fly in 'compare' and 'categorize'

[olgabot]

I'd like to be able to calculate ONE "deep" signature of e.g. 10k kmers for each sample and then adjust the scaling factor to 5k kmers, 1k kmers, 500 kmers, 100 kmers, see how that affects the similarity and nearest neighbors.

[ctb]

Have you tried using --scaled with search and compare?

[olgabot]

Thanks for the quick response! My issue is that droplet-based methods (~10k reads/cell) are much shallower than full-transcript methods (~500k reads/cell) and I'd like to use the same number of kmers/hashes across both, not just the same scaling factor.

[ctb]

On Fri, Oct 26, 2018 at 04:35:17PM +0000, Olga Botvinnik wrote: Thanks for the quick response! My issue is that droplet-based methods (~10k reads/cell) are much shallower than full-transcript methods (~500k reads/cell) and I'd like to use the same number of kmers/hashes across both, not just the same scaling factor.

my intuition is that neither --num nor --scaled will solve your concern here, because you simply have more k-mers to look at either way and neither method is robust to that. I think you probably want to downsample your data by number of reads instead. but that's off the cuff and could well be wrong :)

[luizirber]

would something like #538 help?

[olgabot]

@luizirber Yes, it would be very helpful to be able to switch between --scaled and --num-hashes! I have many many (10/cell, 1000 cells) that are essentially redundant because they used different scales and I didn't notice the --scaled feature in compare before.

[ctb]

this can be done programmatically quite easily (construct a new MinHash object, shove hashes in with add_many). I'm more skeptical that this needs to be supported by sourmash at the command-line level, or at least I feel like it would add significantly to the complexity. What do you think about us providing an example script that does the right thing, without integrating it into the command line?

[olgabot]

that would be very helpful! and definitely eliminate the number of redundant computations I'm doing

[ctb]

sorry for dropping the ball on this...

but, looking at #587, I wonder if this functionality fits under downsample? I could imagine adding a flag to the command like so:

sourmash signature downsample --num 500 file1.sig

and/or

sourmash signature downsample --scaled 1000 file1.sig

where --num and --scaled are (for the moment) incompatible, and fail when the downsampling cannot be done properly. Whaddya think?

[ctb]

Also, in #436 I added some example Python code :)

[ctb]

#1072 would resolve this, I think.

[ctb]

The core functionality is available in sourmash sig downsample, and the selector framework discussed in #1524 is probably the right place to implement something generic in the future.

Closing until the specific CLI functionality is requested again :)