lotus2 DB issues

[paulzierep]

It seems that required DBs for lotus are missing on the EU Galaxy instance:

WARNING:: Requested DB for uchime ref at
DB//ITS_chimera/uchime_sh_refs_dynamic_original_985_11.03.2015.fasta
does not exist; LotuS will run without reference based OTU chimera checking.
Could not find UNITE ITS DB files at 
DB//UNITE/sh_refs_qiime_ver8_99_s_all_02.02.2019.fasta
DB//UNITE/sh_taxonomy_qiime_ver8_99_s_all_02.02.2019.txt
Please check that these files exist.

Does it use a DataManager ? Or are the DBs installed when the tool is installed via conda ?

[paulzierep]

OK, so the DB is not installed if the tool is installed in a docker container: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/lotus2/post-link.sh; any reason for this ? Keeping the container small ? What would be the ideal solution, (a) install when the wrapper is executed, that will take a while. (B) write a Data Manager and link the DB for the wrapper ?

[bgruening]

Ah, ok, I have followed this discussion. I know why ... too big for the CI. Can we run this tool in Conda for the time being?

[paulzierep]

If it is possible to run individual tools in conda on EU that would probably solve it, anyway I can support this?

[nsoranzo]

Sorry for the late reply, busy with too many things (including lotus2 2.30 update).
Yes, the lotus2 biocontainer cannot currently contain the databases because the container building fails on the bioconda CI with "No space left on device", see bioconda/bioconda-recipes#41675

The short-term solution is indeed to use the conda package, which installs some databases at install time. After that, to have the full set of indexes you'd need to manually run the following:

# Activate the lotus2 conda env
$ source YOUR_PATH_TO/miniconda3/bin/activate __lotus2@2.32
$ cd YOUR_PATH_TO/miniconda3/envs/__lotus2@2.32/share/lotus2-2.32-0/
$ perl autoInstall.pl -condaDBinstall -downloadLmbdIdx

which takes some hours and needs to be done on a machine which has at least 32 GB of memory.

Next steps on our side:

• Update the Galaxy wrapper to lotus2 2.30, which replaces lambda1 with lambda3 (much faster in indexing, but still uses a lot of memory)
• Provide downloadable lambda3 indexes
• Add an option to lotus2 so it can run with the databases installed in a different directory (currently everything is specified in lOTUs.cfg
• Add a Galaxy data manager to install data for a specific lotus2 version.

[bgruening]

It should work now on EU. Please try.

[bgruening]

xref: galaxyproject/tpv-shared-database#52

[paulzierep]

the tool runs but fails with:

Warnings for mapping file 
/data/jwd02f/main/065/598/65598385/configs/tmpk1gfmb5i 
Abort by pressing Ctrl+c (10 sec wait)
*********
--------------------------------------------------------------------------------
 00:00:12 Reading mapping file
          Sequence files are indicated in mapping file.
          Found "SequencingRun" column, with 1 categories (a)
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
 00:00:12 Demultiplexing, filtering, dereplicating input files, this
          might take some time..
          check progress at output/LotuSLogS/LotuS_progout.log
 00:00:12 Finished primary read processing with sdm:
          Reads processed: 124
          Accepted (High qual): 4 (45 end-trimmed)
          Accepted (Mid qual): 1
          Rejected: 119
          Dereplication block 0: 0 unique sequences (avg size -nan; 0 counts)
          For an extensive report see output/LotuSLogS//demulti.log
--------------------------------------------------------------------------------
The sdm dereplicated output file was either empty or not existing, aborting lotus.
tmp_folder/derep.fas

%@#%@#%@#%@%@#@%#@%#@#%@#%@#%@#@%#@%#@%#@#%@#%@#%@##
      LotuS2 encounterend an error:
The sdm dereplicated output file was either empty or not existing, aborting lotus.
tmp_folder/derep.fas

There was no demultiplex file provided.

[nsoranzo]

@paulzierep Can you share your history?

This was resolved offline.