inconsistencies in pathlist and directory loading with MultiIndex

[ctb]

ohmigod this is killing me... 😭

When loading files, pathlists and directories in sourmash, we use MultiIndex.

For loading a pathlist, we use MultiIndex.load_from_pathlist. This will load any kind of index object from a text file containing a list of paths.

When loading a path, we use MultiIndex.load_from_path.

• if the path is a directory, we call MultiIndex.load_from_directory. This does a traversal of the directory hierarchy and tries to load all files using a LinearIndex, i.e. as a sig/sig.gz file.
• if the path is a single file that is a sig/sig.gz file, we load it using LinearIndex.

In particular this means that we can load all kinds of things (zip files, in particular, but also SBTs, LCAs, SQLite databases, or really anything for which we have a plugin) from pathlists. From directory hierarchies, however, we can only load sig/sig.gz files.

This seems frustrating and inconsistent.

Related:

• sig collect should resolve to individual files when loading a pathlist or a directory #3039

[ctb]

I think the solution is to just recommend against pathlist and directory loading in sourmash overall, and suggest that people use --from-file if they have lots of files, and/or standalone manifests, and/or zipfiles. Viz #3027.