can we make some of the 'sourmash signature' functions work on streams?

[ctb]

ref #587. e.g. sourmash sig describe could certainly be made streaming; would be interesting to explore for other commands, as an alternative solution to adding detailed/specific command line parameters as in e.g. #560.

[ctb]

thinking more about this, what I'm really after is chaining (ref).

In particular, I'd really like to be able to do something like this:

<trim command that outputs sequences to stdout> | sourmash monitor -o sig1 |
    <another trim/whatever command> | sourmash monitor -o sig2 |
    <streaming assembler> | sourmash monitor -o sig3

which would let us watch loss of k-mers etc from these commands. (I'm already using this kind of thing in spacegraphcats to monitor information loss at various steps.)

More specifically to this issue, I have visions of

sourmash sig extract --md5=... -k 31 |
     sourmash sig rename foo - |
     sourmash sig downsample -o search.sig

or even more ambitiously

sourmash sig extract <list of gather results> |
   sourmash compare -o foo -

• basically, be able to bypass a bunch of temp files and do various powerfully things on the fly.

cc @standage

[standage]

To be clear, you're suggesting a tube that takes in a stream of sequences, maintains a minhash sketch of incoming k-mers, and spits out each sequence unmodified? Sounds super useful, and (although I know little about how sourmash is implemented) should be straightforward I would think.

If you wanted to fully support a continuous data stream, you'd probably want to write the minhash sketch to disk at regular intervals. But I doubt that use case is in high demand just yet. :-)

[ctb]

see also: sourmash watch, which does essentially this :)

[ctb]

I think this was fixed by #1059, which fixed #1049 ... confirmation:

sourmash sig cat tests/test-data/prot/protein/GCA_001593935.1_ASM159393v1_protein.faa.gz.sig tests/test-data/prot/protein/GCA_001593935.1_ASM159393v1_protein.faa.gz.sig | sourmash compare - -o xxx

works just fine.