[MRG] implement sourmash sketch fromfile

doc/README.md

@@ -0,0 +1,18 @@
+# Documentation on the docs
+
+We use
+[MyST](https://myst-parser.readthedocs.io/en/latest/sphinx/intro.html)
+to generate Sphinx doc output from Markdown input.
+
+## Useful tips and tricks:
+
+### Linking internally between sections in the docs
+
+For linking within the sourmash docs, you should use the
+[auto-generated header anchors](https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#auto-generated-header-anchors)
+provided by MyST.
+
+You can generate a list of these for a given document with:
+```
+myst-anchors -l 3 command-line.md
+```

doc/command-line.md

@@ -119,12 +119,13 @@ information for each command.
 
 Most of the commands in sourmash work with **signatures**, which contain information about genomic or proteomic sequences. Each signature contains one or more **sketches**, which are compressed versions of these sequences. Using sourmash, you can search, compare, and analyze these sequences in various ways.
 
-To create a signature with one or more sketches, you use the `sourmash sketch` command. There are three main commands:
+To create a signature with one or more sketches, you use the `sourmash sketch` command. There are four main commands:
 
 ```
 sourmash sketch dna
 sourmash sketch protein
 sourmash sketch translate
+sourmash sketch fromfile
 ```
 
 The `sketch dna` command reads in **DNA sequences** and outputs **DNA sketches**.
@@ -133,10 +134,14 @@ The `sketch protein` command reads in **protein sequences** and outputs **protei
 
 The `sketch translate` command reads in **DNA sequences**, translates them in all six frames, and outputs **protein sketches**.
 
-`sourmash sketch` takes FASTA or FASTQ sequences as input; input data can be
-uncompressed, compressed with gzip, or compressed with bzip2. The output
-will be one or more JSON signature files that can be used with the other
-sourmash commands.
+The `sketch fromfile` command takes in a CSV file containing the
+locations of genomes and proteomes, and outputs all of the requested
+sketches. It is primarily intended for large-scale database construction.
+
+All of the `sourmash sketch` commands take FASTA or FASTQ sequences as
+input; input data can be uncompressed, compressed with gzip, or
+compressed with bzip2. The output will be one or more signature files
+that can be used by other sourmash commands.
 
 Please see
 [the `sourmash sketch` documentation page](sourmash-sketch.md) for
@@ -1585,6 +1590,9 @@ to stdout.
 
 All of these save formats can be loaded by sourmash commands.
 
+**We strongly suggest using .zip files to store signatures: they are fast,
+small, and fully supported by all the sourmash commands.**
+
 ### Loading many signatures
 
 #### Loading signatures within a directory hierarchy

doc/developer.md

@@ -105,6 +105,11 @@ Code coverage can be viewed interactively at [codecov.io][1].
 [1]: https://codecov.io/gh/sourmash-bio/sourmash/
 [2]: https://github.com/sourmash-bio/sourmash/actions
 
+## Writing docs.
+
+Please see [the docs README](README.md) for information on how we
+write and build the sourmash docs.
+
 ## Code organization
 
 There are three main components in the sourmash repo:

doc/sourmash-sketch.md

@@ -1,5 +1,9 @@
 # `sourmash sketch` documentation
 
+```{contents} Contents
+:depth: 3
+```
+
 Most of the commands in sourmash work with **signatures**, which contain information about genomic or proteomic sequences. Each signature contains one or more **sketches**, which are compressed versions of these sequences. Using sourmash, you can search, compare, and analyze these sequences in various ways.
 
 To create a signature with one or more sketches, you use the `sourmash sketch` command. There are three main commands:
@@ -8,6 +12,7 @@ To create a signature with one or more sketches, you use the `sourmash sketch` c
 sourmash sketch dna
 sourmash sketch protein
 sourmash sketch translate
+sourmash sketch fromfile
 ```
 
 The `sketch dna` command reads in **DNA sequences** and outputs **DNA sketches**.
@@ -16,10 +21,14 @@ The `sketch protein` command reads in **protein sequences** and outputs **protei
 
 The `sketch translate` command reads in **DNA sequences**, translates them in all six frames, and outputs **protein sketches**.
 
+The `sketch fromfile` command takes in a CSV file containing the
+locations of genomes and proteomes, and outputs all of the requested
+sketches. It is primarily intended for large-scale database construction.
+
 All `sourmash sketch` commands take FASTA or FASTQ sequences as input;
 input data can be uncompressed, compressed with gzip, or compressed
-with bzip2. The output will be one or more JSON signature files that
-can be used with the other sourmash commands.
+with bzip2. The output will be one or more signature files that
+can be used by other sourmash commands.
 
 ## Quickstart
 
@@ -61,6 +70,53 @@ If you want to use different encodings, you can specify them in a few ways; here
 sourmash sketch protein -p k=25,scaled=500,dayhoff genome.faa
 ```
 
+### Translated DNA sketches for metagenomes
+
+The command
+```
+sourmash sketch translate metagenome.fq
+```
+will take each read in the FASTQ file and translate the read into
+amino acid sequence in all six possible coding frames. No attempt is
+made to determine the right frame (but we are working on ways to
+determine this; see [orpheum](https://github.com/czbiohub/orpheum)).
+
+We suggest using this primarily on unassembled metagenome data. For
+most microbial genomes, it is both higher quality and more efficient
+to first predict the coding sequences (using e.g. prodigal) and then
+use `sketch protein` to build signatures.
+
+### Bulk sketch construction from many files
+
+The `sourmash sketch fromfile` command is intended for use when
+building many signatures as part of a larger workflow. It supports a
+variety of options to build new signatures, parallelize
+signature construction, and otherwise aid in tracking and managing
+database construction.
+
+The command
+```
+sourmash sketch fromfile datasets.csv -p dna -p protein -o database.zip
+```
+will ingest a CSV spreadsheet containing (at a minimum) the three columns
+`name`, `genome_filename`, and `protein_filename`, and build all of
+the signatures requested by the parameter strings.  Other columns in
+this file will be ignored.
+
+If no protein, hp, or dayhoff sketches are requested, `protein_filename`
+can be empty for a given row; likewise, if no DNA sketches are requested,
+`genome_filename` can be empty for a given row.
+
+Some of the key command-line options supported by `fromfile` are:
+* `-o/--output-signatures` will save generated signatures to any of the [standard supported output formats](command-line.md#saving-signatures-more-generally).
+* `-o/--output-csv-info` will save a CSV file of input filenames and parameter strings for use with the `sourmash sketch` command line; this can be used to construct signatures in parallel.
+* `--already-done` will take a list of existing signatures/databases to check against; signatures with matching names and parameter strings will not be rebuilt.
+* `--output-manifest-matching` will output a manifest of already-existing signatures, which can then be used with `sourmash sig cat` to collate signatures across databases; see [using manifests](command-line.md#using-manifests-to-explicitly-refer-to-collections-of-files). (This provides [`sourmash sig check` functionality](command-line.md#sourmash-signature-check---compare-picklists-and-manifests) in `sketch fromfile`.)
+
+If you would like help and advice on constructing large databases, or
+pointers to code for generating the `fromfile` CSV format, please ask
+[on the sourmash issue tracker](https://github.com/sourmash-bio/sourmash/issues) or [gitter support channel](https://gitter.im/sourmash-bio/community).
+
 ## More detailed documentation
 
 ### Input formats
@@ -189,7 +245,7 @@ Unfortunately, changing the k-mer size or using different DNA/protein encodings
 
 ### Examining the output of `sourmash sketch`
 
-You can use `sourmash sig describe` to get detailed information about the contents of a signature file. This can help if you want to see exactly what a particular `sourmash sketch` command does!
+You can use `sourmash sig describe` to get detailed information about the contents of a signature file, and `sourmash sig fileinfo` to get a human-readable summary of the contents. This can help if you want to see exactly what a particular `sourmash sketch` command does!
 
 ### Filing issues and asking for help
 

src/sourmash/cli/sketch/__init__.py

@@ -10,6 +10,7 @@
 from . import protein as aa
 from . import protein as prot
 from . import translate
+from . import fromfile
 from ..utils import command_list
 from argparse import SUPPRESS, RawDescriptionHelpFormatter
 import os

src/sourmash/cli/sketch/dna.py

@@ -38,7 +38,7 @@ def subparser(subparsers):
     )
     subparser.add_argument(
         '--check-sequence', action='store_true',
-        help='complain if input sequence is invalid (NOTE: only checks DNA)'
+        help='complain if input sequence is invalid DNA'
     )
     subparser.add_argument(
         '-p', '--param-string', default=[],

src/sourmash/cli/sketch/fromfile.py

@@ -0,0 +1,78 @@
+"""create signatures from a CSV file"""
+
+usage="""
+
+    sourmash sketch fromfile <csv file> --output-signatures <location> -p <...>
+
+The 'sketch fromfile' command takes in a CSV file with list of names
+and filenames to be used for building signatures. It is intended for
+batch use, when building large collections of signatures.
+
+One or more parameter strings must be specified with '-p'.
+
+One or more existing collections of signatures can be provided via
+'--already-done' and already-existing signatures (based on name and
+sketch type) will not be recalculated or output.
+
+If a location is provided via '--output-signatures', signatures will be saved
+to that location.
+
+Please see the 'sketch' documentation for more details:
+  https://sourmash.readthedocs.io/en/latest/sourmash-sketch.html
+"""
+
+import sourmash
+from sourmash.logging import notify, print_results, error
+
+from sourmash import command_sketch
+
+
+def subparser(subparsers):
+    subparser = subparsers.add_parser('fromfile',
+                                      usage=usage)
+    subparser.add_argument(
+        'csvs', nargs='+',
+        help="input CSVs providing 'name', 'genome_filename', and 'protein_filename'"
+    )
+    subparser.add_argument(
+        '-p', '--param-string', default=[],
+        help='signature parameters to use.', action='append',
+    )
+    subparser.add_argument(
+        '--already-done', nargs='+', default=[],
+        help='one or more collections of existing signatures to avoid recalculating'
+    )
+    subparser.add_argument(
+        '--license', default='CC0', type=str,
+        help='signature license. Currently only CC0 is supported.'
+    )
+    subparser.add_argument(
+        '--check-sequence', action='store_true',
+        help='complain if input sequence is invalid (NOTE: only checks DNA)'
+    )
+    file_args = subparser.add_argument_group('File handling options')
+    file_args.add_argument(
+        '-o', '--output-signatures',
+        help='output computed signatures to this file',
+    )
+    file_args.add_argument(
+        '--force-output-already-exists', action='store_true',
+        help='overwrite/append to --output-signatures location'
+    )
+    file_args.add_argument(
+        '--ignore-missing', action='store_true',
+        help='proceed with building possible signatures, even if some input files are missing'
+    )
+    file_args.add_argument(
+        '--output-csv-info',
+        help='output information about what signatures need to be generated'
+    )
+    file_args.add_argument(
+        '--output-manifest-matching',
+        help='output a manifest file of already-existing signatures'
+    )
+
+
+def main(args):
+    import sourmash.command_sketch
+    return sourmash.command_sketch.fromfile(args)

src/sourmash/cli/sketch/protein.py

@@ -36,10 +36,6 @@ def subparser(subparsers):
         '--license', default='CC0', type=str,
         help='signature license. Currently only CC0 is supported.'
     )
-    subparser.add_argument(
-        '--check-sequence', action='store_true',
-        help='complain if input sequence is invalid'
-    )
     subparser.add_argument(
         '-p', '--param-string', default=[],
         help='signature parameters to use.', action='append',

src/sourmash/cli/sketch/translate.py

@@ -38,7 +38,7 @@ def subparser(subparsers):
     )
     subparser.add_argument(
         '--check-sequence', action='store_true',
-        help='complain if input sequence is invalid'
+        help='complain if input sequence is invalid DNA'
     )
     subparser.add_argument(
         '-p', '--param-string', default=[],

src/sourmash/command_compute.py

@@ -14,6 +14,7 @@
 from ._lowlevel import ffi, lib
 
 DEFAULT_COMPUTE_K = '21,31,51'
+DEFAULT_MMHASH_SEED = 42
 DEFAULT_LINE_COUNT = 1500
 
 
@@ -197,8 +198,13 @@ def _compute_individual(args, signatures_factory):
             if args.singleton:
                 for n, record in enumerate(screed_iter):
                     sigs = signatures_factory()
-                    add_seq(sigs, record.sequence,
-                            args.input_is_protein, args.check_sequence)
+                    try:
+                        add_seq(sigs, record.sequence,
+                                args.input_is_protein, args.check_sequence)
+                    except ValueError as exc:
+                        error(f"ERROR when reading from '{filename}' - ")
+                        error(str(exc))
+                        sys.exit(-1)
 
                     set_sig_name(sigs, filename, name=record.name)
                     save_sigs_to_location(sigs, save_sigs)
@@ -211,7 +217,7 @@ def _compute_individual(args, signatures_factory):
                 sigs = signatures_factory()
 
                 # consume & calculate signatures
-                notify('... reading sequences from {}', filename)
+                notify(f'... reading sequences from {filename}')
                 name = None
                 for n, record in enumerate(screed_iter):
                     if n % 10000 == 0:
@@ -220,8 +226,13 @@ def _compute_individual(args, signatures_factory):
                         elif args.name_from_first:
                             name = record.name
 
-                    add_seq(sigs, record.sequence,
-                            args.input_is_protein, args.check_sequence)
+                    try:
+                        add_seq(sigs, record.sequence,
+                                args.input_is_protein, args.check_sequence)
+                    except ValueError as exc:
+                        error(f"ERROR when reading from '{filename}' - ")
+                        error(str(exc))
+                        sys.exit(-1)
 
                 notify('...{} {} sequences', filename, n, end='')
 
@@ -348,12 +359,11 @@ def from_manifest_row(cls, row):
         else:
             ksize = row['ksize'] * 3
 
-        p = cls([ksize], 42, is_protein, is_dayhoff, is_hp, is_dna,
+        p = cls([ksize], DEFAULT_MMHASH_SEED, is_protein, is_dayhoff, is_hp, is_dna,
                 row['num'], row['with_abundance'], row['scaled'])
 
         return p
 
-
     def to_param_str(self):
         "Convert object to equivalent params str."
         pi = []
@@ -388,7 +398,7 @@ def to_param_str(self):
             pi.append("abund")
         # noabund is default
 
-        if self.seed != 42:
+        if self.seed != DEFAULT_MMHASH_SEED:
             pi.append(f"seed={self.seed}")
         # self.seed
 
@@ -474,6 +484,16 @@ def dna(self):
     def dna(self, v):
         return self._methodcall(lib.computeparams_set_dna, v)
 
+    @property
+    def moltype(self):
+        if self.dna: moltype = 'DNA'
+        elif self.protein: moltype = 'protein'
+        elif self.hp: moltype = 'hp'
+        elif self.dayhoff: moltype = 'dayhoff'
+        else: assert 0
+
+        return moltype
+
     @property
     def num_hashes(self):
         return self._methodcall(lib.computeparams_num_hashes)