HTTP 404 NOT FOUND for channel conda-base <https://conda.anaconda.org/conda-base>

[karlaarz]

Description of the bug

Hi! I've been trying to run the latest version of this pipeline: 2.3.1 using conda and I encountered this error:

ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /net/192.168.120.240/home/tigem/k.ruiz/work/conda/env-f6207f68e2d2d68777c6463d18984735 mirtop=0.4.25 bioconda::samtools=1.15.1 conda-base::r-base=4.1.1 conda-base::r-data.table=1.14.2
  status : 1
  message:
    Channels:
     - defaults
     - bioconda
     - conda-base
    Platform: linux-64
    Collecting package metadata (repodata.json): ...working... failed
    
    UnavailableInvalidChannel: HTTP 404 NOT FOUND for channel conda-base <https://conda.anaconda.org/conda-base>
    
    The channel is not accessible or is invalid.
    
    You will need to adjust your conda configuration to proceed.
    Use `conda config --show channels` to view your configuration's current state,
    and use `conda config --show-sources` to view config file locations.

I asked at conda's Github conda/conda#13823 and was suggested to change conda-base to conda-forge instead. How can I do this?

Command used and terminal output

nextflow run nf-core/smrnaseq -r 2.3.1 \
	-profile conda \
	--input 'small_PB.csv' \
	--fasta Mus_musculus.GRCm39.dna.toplevel.fa \
	--mirtrace_species 'mmu' \
	--mirna_gtf mmu.gff3 \
	--hairpin hairpin.fa \
	--mature mature.fa \
	--protocol 'illumina' \
	--outdir v23 \
	-resume

Relevant files

nextflow.log

System information

No response

[christopher-mohr]

Hi @karlaarz, thanks for reporting! Could you give it another try with the dev branch?

[karlaarz]

Hi @christopher-mohr, sorry for my late response. I tried with the dev branch, and the conda problem is solved now, however, I encountered another error:

Error executing process > 'NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT'

Caused by:
  Process `NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT` terminated with an error exit status (1)

Command executed:

  #Cleanup the GTF if mirbase html form is broken
  GTF="mmu.gff3"
  sed 's/>/>/g' $GTF | sed 's#
#\n#g' | sed 's#
##g' | sed 's#
##g' | sed -e :a -e '/^\n*$/{$d;N;};/\n$/ba' > ${GTF}_html_cleaned.gtf
  mirtop gff --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf -o mirtop --sps mmu ./bams/*
  mirtop counts --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf -o mirtop --sps mmu --add-extra --gff mirtop/mirtop.gff
  mirtop export --format isomir --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf --sps mmu -o mirtop mirtop/mirtop.gff
  mirtop stats mirtop/mirtop.gff --out mirtop/stats
  mv mirtop/stats/mirtop_stats.log mirtop/stats/full_mirtop_stats.log
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT":
      mirtop: $(echo $(mirtop --version 2>&1) | sed 's/^.*mirtop //')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/bin/mirtop", line 6, in 
      from mirtop.command_line import main
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/command_line.py", line 8, in 
      from mirtop.gff import reader
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/gff/__init__.py", line 7, in 
      from mirtop.bam.bam import read_bam
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/bam/bam.py", line 7, in 
      import pysam
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/pysam/__init__.py", line 5, in 
      from pysam.libchtslib import *
  ImportError: libhts.so.2: cannot open shared object file: No such file or directory

Work dir:
  /net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/c4/454c9050eebae776beb1138c6e4564

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Thanks for your help!

[karlaarz]

I also tried adding the database argument in the config file (#329), but I still got the same error.

process {
        withName: 'MIRTOP_QUANT' {
        ext.args = "--database miRBase"
    }
}

[apeltzer]

What type of FS is behind this? /net/192.168.120.240/home/ that looks a bit like a smb/nfs network mount. Also the libhts.so.2 error suggests that something is missing in your environment. Can you try running with singularity or docker? Conda is sometimes problematic...

[christopher-mohr]

Hi @christopher-mohr, sorry for my late response. I tried with the dev branch, and the conda problem is solved now, however, I encountered another error:

Error executing process > 'NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT'

Caused by:
  Process `NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT` terminated with an error exit status (1)

Command executed:

  #Cleanup the GTF if mirbase html form is broken
  GTF="mmu.gff3"
  sed 's/>/>/g' $GTF | sed 's#
#\n#g' | sed 's#
##g' | sed 's#
##g' | sed -e :a -e '/^\n*$/{$d;N;};/\n$/ba' > ${GTF}_html_cleaned.gtf
  mirtop gff --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf -o mirtop --sps mmu ./bams/*
  mirtop counts --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf -o mirtop --sps mmu --add-extra --gff mirtop/mirtop.gff
  mirtop export --format isomir --hairpin hairpin.fa_igenome.fa_idx.fa --gtf ${GTF}_html_cleaned.gtf --sps mmu -o mirtop mirtop/mirtop.gff
  mirtop stats mirtop/mirtop.gff --out mirtop/stats
  mv mirtop/stats/mirtop_stats.log mirtop/stats/full_mirtop_stats.log
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SMRNASEQ:MIRNA_QUANT:MIRTOP_QUANT":
      mirtop: $(echo $(mirtop --version 2>&1) | sed 's/^.*mirtop //')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/bin/mirtop", line 6, in 
      from mirtop.command_line import main
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/command_line.py", line 8, in 
      from mirtop.gff import reader
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/gff/__init__.py", line 7, in 
      from mirtop.bam.bam import read_bam
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/mirtop/bam/bam.py", line 7, in 
      import pysam
    File "/net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/conda/env-6ac1df4e005f7cb6e73884f868896d31/lib/python3.6/site-packages/pysam/__init__.py", line 5, in 
      from pysam.libchtslib import *
  ImportError: libhts.so.2: cannot open shared object file: No such file or directory

Work dir:
  /net/192.168.120.240/home/tigem/k.ruiz/paola/small/v23/work/c4/454c9050eebae776beb1138c6e4564

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Thanks for your help!

I can reproduce the issue on GitPod. Looks like something might be wrong with the conda package, more specificially pysam: libhts.so.2

As @apeltzer suggested, if possible it would be better to use singularity or docker for now.

I will still try to fix the issue with conda. It might help to use a more recent python version.

[karlaarz]

Thanks, @christopher-mohr and @apeltzer, for your replies. I updated my Python version, but is still the same error.

The /net/192.168.120.240/home/ is because I'm working on an HPC cluster. I've tried using singularity and setting up the environment variables in my bash profile according to the cluster manage:

export NXF_SINGULARITY_CACHEDIR="/home/tigem/k.ruiz/paola/small/sing"
export SINGULARITY_CACHEDIR="/home/tigem/k.ruiz/paola/small/sing"
export SINGULARITY_TMPDIR="/home/tigem/k.ruiz/paola/small/sing"

But I get this error:

ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:FASTQ_FASTQC_UMITOOLS_FASTP:FASTP (6)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img.pulling.1715013113601 https://depot.galaxyproject.org/singularity/fastp:0.23.4--h5f740d0_0 > /dev/null
  status : 255
  message:
    ERROR: pull is only supported for shub URIs

I completely cleared the ~/.singularity directory and ran it again, but it still does not work; I have almost 500GB of free space, so definitely is not problem of space.

[maxulysse]

What is your singularity version?

[karlaarz]

It is singularity-ce version 4.0.3

[christopher-mohr]

Thanks, @christopher-mohr and @apeltzer, for your replies. I updated my Python version, but is still the same error.

Sorry @karlaarz, I should have been more precise. I meant the Python version in the conda env. I changed it accordingly and ran a test succesfully. Could you please try it again with the updated dev branch?

[karlaarz]

Hi @christopher-mohr, yes, the Python's version of my conda is the 3.12.3. I tried it again but still got the same error.

[christopher-mohr]

Hi @christopher-mohr, yes, the Python's version of my conda is the 3.12.3. I tried it again but still got the same error.

Hi @karlaarz, did you run the following before re-running the pipeline?
nextflow pull nf-core/smrnaseq -r dev

[karlaarz]

Hi @christopher-mohr , you are right! After I did nextflow pull nf-core/smrnaseq -r dev, the pipeline ended correctly. Thanks a lot :D

[christopher-mohr]

Hi @christopher-mohr , you are right! After I did nextflow pull nf-core/smrnaseq -r dev, the pipeline ended correctly. Thanks a lot :D

Nice to hear! :) I would then close this issue. Thanks again for reporting! If you want to follow-up on your Singularity issue I would suggest to post it on Slack.