bp-assembly phase failing at 5-phase/index-pair

[danagibbon]

Operating system
Which operating system and version are you using?

CentOS Linux release 7.5.1804

Package name
Which package / tool is causing the problem? Which version are you using, use tool --version. Have you updated to the latest version conda update package? Have you updated the complete env by running conda update --all? Have you ensured that your channel priorities are set up according to the bioconda recommendations at https://bioconda.github.io/#set-up-channels?

• I'm having an issue with bp-assembly's fc_phase.py on all datasets, including the F1_bull_test
• I've run conda update --all & conda update pb-assembly and get: All requested packages already installed.

Conda environment
What is the result of conda list? (Try to paste that between triple backticks.)

• Output from conda list:

 # Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
bcftools                  1.9                  ha228f0b_4    bioconda
bedtools                  2.28.0               hdf88d34_0    bioconda
blas                      1.0                         mkl  
blasr                     5.3.3                h707fff8_0    bioconda
blasr_libcpp              5.3.3                h707fff8_0    bioconda
bwa                       0.7.17               h84994c4_5    bioconda
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2019.5.15                     0  
certifi                   2019.6.16                py37_0  
curl                      7.65.2               hbc83047_0  
decorator                 4.4.0                    py37_1  
falcon-kit                1.4.2                    pypi_0    pypi
falcon-phase              1.1.0                    pypi_0    pypi
falcon-unzip              1.3.3                    pypi_0    pypi
future                    0.17.1                   py37_0  
hdf5                      1.10.4               hb1b8bf9_0  
htslib                    1.9                  ha228f0b_7    bioconda
intel-openmp              2019.4                      243  
krb5                      1.16.1               h173b8e3_7  
libcurl                   7.65.2               h20c2e04_0  
libdeflate                1.0                  h14c3975_1    bioconda
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libssh2                   1.8.2                h1ba5d50_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
minimap2                  2.17                 h84994c4_0    bioconda
mkl                       2019.4                      243  
mkl_fft                   1.0.12           py37ha843d7b_0  
mkl_random                1.0.2            py37hd81dba3_0  
mummer4                   4.0.0beta2      pl526hf484d3e_4    bioconda
ncurses                   6.1                  he6710b0_1  
networkx                  2.3                        py_0  
nim-falcon                0.0.0                         0    bioconda
numpy                     1.16.4           py37h7e9f1db_0  
numpy-base                1.16.4           py37hde5b4d6_0  
openssl                   1.1.1c               h7b6447c_1  
pb-assembly               0.0.6                    py37_5    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.3.3            py37hc0aa232_0    bioconda
pbbam                     0.23.0               h88345a9_0    bioconda
pbcopper                  0.4.2                h3f2a3d2_1    bioconda
pbgcpp                    0.0.1                         0    bioconda
pbmm2                     1.0.0                ha888412_0    bioconda
perl                      5.26.2               h14c3975_0  
pip                       19.1.1                   py37_0  
pysam                     0.15.2           py37h4b7d16d_3    bioconda
python                    3.7.3                h0371630_0  
python-edlib              1.2.4.post1      py37h6bb024c_0    bioconda
python-intervaltree       3.0.2                      py_0    bioconda
python-msgpack            0.6.1            py37h6bb024c_0    bioconda
python-sortedcontainers   2.1.0                      py_0    bioconda
racon                     1.4.0                he860b03_0    bioconda
readline                  7.0                  h7b6447c_5  
samtools                  1.9                 h8571acd_11    bioconda
setuptools                41.0.1                   py37_0  
sqlite                    3.29.0               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wheel                     0.33.4                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3

Describe the bug
A clear and concise description of what the bug is.

• I was able to successfully run falcon and unzip, but get an error when I run falcon phase
• I am using the F1_bull_test_data set

Error message
Paste the error message / stack.

falcon-phase 1.1.0
falcon-kit 1.4.2
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]Using config=
{'General': {},
 'Phase': {'cns_h_ctg_fasta': './4-polish/cns-output/cns_h_ctg.fasta',
           'cns_p_ctg_fasta': './4-polish/cns-output/cns_p_ctg.fasta',
           'enzyme': '"GATC"',
           'iterations': '10000000',
           'min_aln_len': '5000',
           'output_format': 'pseudohap',
           'reads_1': '/lustre/hpcprod/skingan/FALCON_tutorial/f1_bull/F1_bull_test_data/F1_bull_test.HiC_R1.fastq.gz',
           'reads_2': '/lustre/hpcprod/skingan/FALCON_tutorial/f1_bull/F1_bull_test_data/F1_bull_test.HiC_R2.fastq.gz'},
 'job.defaults': {'MB': '16000',
                  'NPROC': '8',
                  'job_type': 'string',
                  'njobs': '100',
                  'pwatcher_type': 'blocking',
                  'submit': '/bin/bash -c "${JOB_SCRIPT}" > "${JOB_STDOUT}" 2> '
                            '"${JOB_STDERR}"',
                  'use_tmpdir': False},
 'job.step.phase.alignment': {},
 'job.step.phase.placement': {}}
[INFO]PATH=/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin:/nfs3/CGRB/home/gibbond/miniconda3/condabin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/local/cluster/neovim/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2:/nfs3/CGRB/home/gibbond/local/bin/samtools-1.9/bin:/nfs3/CGRB/home/gibbond/local/bin/bcftools-1.9/bin:/local/cluster/R-3.5.0/bin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/home/cgrb/gibbond/scripts:/home/cgrb/gibbond/bin:/local/cluster/jdk1.8.0_71/jre/bin:/bin:/usr/bin:/local/cluster/ncbi-blast+/bin:/local/cluster/bin:/usr/local/bin:/local/cluster/mpich/bin:/usr/local/share/ncbi/bin:/local/cluster/hdf5-1.8.12/bin:/local/cluster/genome/bin:/local/cluster/RECON/bin:/local/cluster/RECON/scripts:/local/cluster/MUMmer:/local/cluster/amos/bin:/local/cluster/velvet/velvet:/local/cluster/oases:/local/cluster/mira/bin:/local/cluster/abyss/bin:/local/cluster/cutadapt/bin:/local/cluster/edena2.1.1_linux64:/local/cluster/MAKER/bin:/local/cluster/mcl/bin:/local/cluster/YASRA/bin:/local/cluster/miRanda/bin:/local/cluster/ea-utils/bin:/local/cluster/RAxML/bin:/local/cluster/MOSAIK/bin:/local/cluster/hmmer/bin:/local/cluster/meme/bin:/local/cluster/tmhmm/bin:/local/cluster/wgs/Linux-amd64/bin:/local/cluster/amber16/bin:/local/cluster/mpich2-1.2.1p1/bin:/usr/lib64/lam/bin:/local/cluster/mockler/bin:/local/cluster/carrington/bin:/local/cluster/variscan-2.0.3/bin/Linux-i386:/local/cluster/Roche/454/bin:/local/cluster/shore:/local/cluster/SHOREmap:/local/cluster/BEAST/bin:/local/cluster/BEDTools/bin:/local/cluster/genomemapper:/local/cluster/iprscan/bin:/local/cluster/trinityrnaseq:/local/cluster/Cerulean/bin:/local/cluster/Quake/bin:/local/cluster/glimmer/bin:/local/cluster/samtools/bin:/local/cluster/SPAdes/bin:/local/cluster/RAPSearch2.16_64bits/bin:/local/cluster/last-418/bin:/local/cluster/rnammer:/local/cluster/SHRiMP/bin:/local/cluster/homer/bin:/local/cluster/cd-hit:/local/cluster/augustus/bin:/local/cluster/structure_linux_console/bin:/local/cluster/stampy:/local/cluster/infernal/binaries:/local/cluster/rtax:/local/cluster/pandaseq/bin:/local/cluster/bedtools2/bin:/local/cluster/htsjdk/dist:/local/cluster/GARM:/local/cluster/AmpliconNoise/ampliconnoise/Scripts:/local/cluster/AmpliconNoise/ampliconnoise:/local/cluster/pplacer-v1.1:/local/cluster/microbiomeutil/WigeoN:/local/cluster/microbiomeutil/TreeChopper:/local/cluster/microbiomeutil/NAST-iEr:/local/cluster/microbiomeutil/ChimeraSlayer:/local/cluster/AmosCmp16Spipeline:/local/cluster/Tisean_3.0.0/bin:/local/cluster/allpathslg/bin:/local/cluster/NAMD:/local/cluster/vcf/bin:/local/cluster/iRODS/clients/icommands/bin:/local/cluster/SVMerge/bin:/local/cluster/pindel/bin:/local/cluster/breakdancer-1.1.2/bin:/local/cluster/cnD/bin:/local/cluster/nextclip/bin:/local/cluster/prokka/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/julia-0.3.3/bin:/local/cluster/jnet/bin:/local/cluster/mongodb/bin:/local/cluster/gsl/bin:/local/cluster/sratoolkit/bin:/local/cluster/wise2.2.3-rc7/src/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/freebayes/bin:/local/cluster/stacks/bin:/local/cluster/PhyloTreePruner:/local/cluster/barrnap-0.6/bin:/local/cluster/glpk/bin:/local/cluster/metaphlan/bin:/local/cluster/SOAPdenovo-Trans/bin:/local/cluster/ruby/bin:/local/cluster/fastqc:/local/cluster/PBSuite_15.8.24/bin:/local/cluster/detonate-1.10/bin:/local/cluster/canu/Linux-amd64/bin:/local/cluster/Blast2GO/bin:/local/cluster/JAGS-4.2.0/bin:/local/cluster/capnproto-c++/bin:/local/cluster/ucsc_genome:/local/cluster/vcflib/bin:/local/cluster/tre/bin:/local/cluster/MaSuRCA/bin:/local/cluster/BUSCO/bin:/local/cluster/BUSCO_v3.02/bin:/local/cluster/bib/active/bin:/local/cluster/Platypus/bin:/local/cluster/rMATS/bin:/local/cluster/sox/bin:/local/cluster/hisat2/bin:/local/cluster/bbmap:/local/cluster/mirdeep2/bin:/local/cluster/miniconda2/bin:/local/cluster/groff-1.22.3/bin:/local/cluster/EPACTS/bin:/local/cluster/percolator/bin:/local/cluster/mafft/bin:/local/cluster/gmap/bin:/local/cluster/BRAKER/bin:/local/cluster/MATLAB/R2018a/bin:/usr/X11R6/bin:/usr/X/bin:.:/local/cluster/dowser/bin:/local/cluster/dowser/bin/linux:/local/cluster/bamtools/bin:/nfs3/CGRB/home/gibbond/lib/edirect:/nfs3/CGRB/home/davised/opt/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2
[INFO]$('which which')
/bin/which
[INFO]$('which samtools')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/samtools
[INFO]$('which nucmer')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/nucmer
[INFO]$('which show-coords')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/show-coords
[INFO]$('which delta-filter')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/delta-filter
[INFO]$('which bwa')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/bwa
[INFO]$('which bedtools')
/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/bedtools
[INFO]$ nucmer --version >
[INFO]nucmer ['4', '0', '0beta2'] is >= 4.0.0
[INFO]$ bwa mem >
[INFO]$ show-coords -h >
[INFO]$ delta-filter -h >
[INFO]$('bedtools --version')
bedtools v2.28.0
[INFO]$ samtools >
[INFO]samtools ['1', '9'] is >= 1.3
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pwatcher/blocking.py'>
[INFO]job_type='string', (default)job_defaults={'NPROC': '8', 'njobs': '100', 'MB': '16000', 'pwatcher_type': 'blocking', 'job_type': 'string', 'submit': '/bin/bash -c "${JOB_SCRIPT}" > "${JOB_STDOUT}" 2> "${JOB_STDERR}"', 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 100; was None
[INFO]This is where FALCON-Phase workflow should be initialized and run.
[INFO]Num unsatisfied: 0, graph: 2
[INFO]Num unsatisfied: 0, graph: 11
[INFO]Num unsatisfied: 6, graph: 21
[INFO]About to submit: Node(5-phase/index-pair)
[INFO]About to submit: Node(5-phase/alignment)
[INFO]Popen: '/bin/bash -c "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pwatcher/mains/job_start.sh" > "/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/run-P078f34dbbca943.bash.stdout" 2> "/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/run-P078f34dbbca943.bash.stderr"'
[INFO]Popen: '/bin/bash -c "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pwatcher/mains/job_start.sh" > "/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/alignment/run-Pa84c1d35dee237.bash.stdout" 2> "/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/alignment/run-Pa84c1d35dee237.bash.stderr"'
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO](slept for another 0.30000000000000004s -- another 2 loop iterations)
[ERROR]Task Node(5-phase/index-pair) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: {Node(5-phase/index-pair)}
[ERROR]ready: set()
        submitted: {Node(5-phase/alignment)}
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin/fc_phase.py", line 11, in <module>
    load_entry_point('falcon-phase==1.1.0', 'console_scripts', 'fc_phase.py')()
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/falcon_phase/mains/start_falcon_phase.py", line 26, in main
    falcon_phase.run(**vars(args))
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/falcon_phase/falcon_phase.py", line 69, in run
    falcon_phase_all(config)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/falcon_phase/falcon_phase.py", line 20, in falcon_phase_all
    tasks_falcon_phase.run_workflow(wf, config)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/falcon_phase/tasks/tasks_falcon_phase.py", line 523, in run_workflow
    wf.refreshTargets()
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(5-phase/index-pair)}

• 5-phase/index-pair/run-P078f34dbbca943.bash.stderr

executable=${PYPEFLOW_JOB_START_SCRIPT}
+ executable=/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/run-P078f34dbbca943.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60

# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
    if [[ "${timeout}" == "0" ]]; then
        echo "timed out waiting for (${executable})"
        exit 77
    fi
    echo "not executable: '${executable}', waiting ${timeout}s"
    sleep 1
    timeout=$((timeout-1))
done
+ [[ ! -x /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/run-P078f34dbbca943.bash ]]

/bin/bash ${executable}
+ /bin/bash /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/run-P078f34dbbca943.bash
+ '[' '!' -d /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair ']'
+ cd /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export PATH=/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin:/nfs3/CGRB/home/gibbond/miniconda3/condabin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/local/cluster/neovim/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2:/nfs3/CGRB/home/gibbond/local/bin/samtools-1.9/bin:/nfs3/CGRB/home/gibbond/local/bin/bcftools-1.9/bin:/local/cluster/R-3.5.0/bin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/home/cgrb/gibbond/scripts:/home/cgrb/gibbond/bin:/local/cluster/jdk1.8.0_71/jre/bin:/bin:/usr/bin:/local/cluster/ncbi-blast+/bin:/local/cluster/bin:/usr/local/bin:/local/cluster/mpich/bin:/usr/local/share/ncbi/bin:/local/cluster/hdf5-1.8.12/bin:/local/cluster/genome/bin:/local/cluster/RECON/bin:/local/cluster/RECON/scripts:/local/cluster/MUMmer:/local/cluster/amos/bin:/local/cluster/velvet/velvet:/local/cluster/oases:/local/cluster/mira/bin:/local/cluster/abyss/bin:/local/cluster/cutadapt/bin:/local/cluster/edena2.1.1_linux64:/local/cluster/MAKER/bin:/local/cluster/mcl/bin:/local/cluster/YASRA/bin:/local/cluster/miRanda/bin:/local/cluster/ea-utils/bin:/local/cluster/RAxML/bin:/local/cluster/MOSAIK/bin:/local/cluster/hmmer/bin:/local/cluster/meme/bin:/local/cluster/tmhmm/bin:/local/cluster/wgs/Linux-amd64/bin:/local/cluster/amber16/bin:/local/cluster/mpich2-1.2.1p1/bin:/usr/lib64/lam/bin:/local/cluster/mockler/bin:/local/cluster/carrington/bin:/local/cluster/variscan-2.0.3/bin/Linux-i386:/local/cluster/Roche/454/bin:/local/cluster/shore:/local/cluster/SHOREmap:/local/cluster/BEAST/bin:/local/cluster/BEDTools/bin:/local/cluster/genomemapper:/local/cluster/iprscan/bin:/local/cluster/trinityrnaseq:/local/cluster/Cerulean/bin:/local/cluster/Quake/bin:/local/cluster/glimmer/bin:/local/cluster/samtools/bin:/local/cluster/SPAdes/bin:/local/cluster/RAPSearch2.16_64bits/bin:/local/cluster/last-418/bin:/local/cluster/rnammer:/local/cluster/SHRiMP/bin:/local/cluster/homer/bin:/local/cluster/cd-hit:/local/cluster/augustus/bin:/local/cluster/structure_linux_console/bin:/local/cluster/stampy:/local/cluster/infernal/binaries:/local/cluster/rtax:/local/cluster/pandaseq/bin:/local/cluster/bedtools2/bin:/local/cluster/htsjdk/dist:/local/cluster/GARM:/local/cluster/AmpliconNoise/ampliconnoise/Scripts:/local/cluster/AmpliconNoise/ampliconnoise:/local/cluster/pplacer-v1.1:/local/cluster/microbiomeutil/WigeoN:/local/cluster/microbiomeutil/TreeChopper:/local/cluster/microbiomeutil/NAST-iEr:/local/cluster/microbiomeutil/ChimeraSlayer:/local/cluster/AmosCmp16Spipeline:/local/cluster/Tisean_3.0.0/bin:/local/cluster/allpathslg/bin:/local/cluster/NAMD:/local/cluster/vcf/bin:/local/cluster/iRODS/clients/icommands/bin:/local/cluster/SVMerge/bin:/local/cluster/pindel/bin:/local/cluster/breakdancer-1.1.2/bin:/local/cluster/cnD/bin:/local/cluster/nextclip/bin:/local/cluster/prokka/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/julia-0.3.3/bin:/local/cluster/jnet/bin:/local/cluster/mongodb/bin:/local/cluster/gsl/bin:/local/cluster/sratoolkit/bin:/local/cluster/wise2.2.3-rc7/src/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/freebayes/bin:/local/cluster/stacks/bin:/local/cluster/PhyloTreePruner:/local/cluster/barrnap-0.6/bin:/local/cluster/glpk/bin:/local/cluster/metaphlan/bin:/local/cluster/SOAPdenovo-Trans/bin:/local/cluster/ruby/bin:/local/cluster/fastqc:/local/cluster/PBSuite_15.8.24/bin:/local/cluster/detonate-1.10/bin:/local/cluster/canu/Linux-amd64/bin:/local/cluster/Blast2GO/bin:/local/cluster/JAGS-4.2.0/bin:/local/cluster/capnproto-c++/bin:/local/cluster/ucsc_genome:/local/cluster/vcflib/bin:/local/cluster/tre/bin:/local/cluster/MaSuRCA/bin:/local/cluster/BUSCO/bin:/local/cluster/BUSCO_v3.02/bin:/local/cluster/bib/active/bin:/local/cluster/Platypus/bin:/local/cluster/rMATS/bin:/local/cluster/sox/bin:/local/cluster/hisat2/bin:/local/cluster/bbmap:/local/cluster/mirdeep2/bin:/local/cluster/miniconda2/bin:/local/cluster/groff-1.22.3/bin:/local/cluster/EPACTS/bin:/local/cluster/percolator/bin:/local/cluster/mafft/bin:/local/cluster/gmap/bin:/local/cluster/BRAKER/bin:/local/cluster/MATLAB/R2018a/bin:/usr/X11R6/bin:/usr/X/bin:.:/local/cluster/dowser/bin:/local/cluster/dowser/bin/linux:/local/cluster/bamtools/bin:/nfs3/CGRB/home/gibbond/lib/edirect:/nfs3/CGRB/home/davised/opt/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2:/bin
+ PATH=/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/bin:/nfs3/CGRB/home/gibbond/miniconda3/condabin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/local/cluster/neovim/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2:/nfs3/CGRB/home/gibbond/local/bin/samtools-1.9/bin:/nfs3/CGRB/home/gibbond/local/bin/bcftools-1.9/bin:/local/cluster/R-3.5.0/bin:/local/cluster/sge/bin:/local/cluster/sge/bin/lx-amd64:/home/cgrb/gibbond/scripts:/home/cgrb/gibbond/bin:/local/cluster/jdk1.8.0_71/jre/bin:/bin:/usr/bin:/local/cluster/ncbi-blast+/bin:/local/cluster/bin:/usr/local/bin:/local/cluster/mpich/bin:/usr/local/share/ncbi/bin:/local/cluster/hdf5-1.8.12/bin:/local/cluster/genome/bin:/local/cluster/RECON/bin:/local/cluster/RECON/scripts:/local/cluster/MUMmer:/local/cluster/amos/bin:/local/cluster/velvet/velvet:/local/cluster/oases:/local/cluster/mira/bin:/local/cluster/abyss/bin:/local/cluster/cutadapt/bin:/local/cluster/edena2.1.1_linux64:/local/cluster/MAKER/bin:/local/cluster/mcl/bin:/local/cluster/YASRA/bin:/local/cluster/miRanda/bin:/local/cluster/ea-utils/bin:/local/cluster/RAxML/bin:/local/cluster/MOSAIK/bin:/local/cluster/hmmer/bin:/local/cluster/meme/bin:/local/cluster/tmhmm/bin:/local/cluster/wgs/Linux-amd64/bin:/local/cluster/amber16/bin:/local/cluster/mpich2-1.2.1p1/bin:/usr/lib64/lam/bin:/local/cluster/mockler/bin:/local/cluster/carrington/bin:/local/cluster/variscan-2.0.3/bin/Linux-i386:/local/cluster/Roche/454/bin:/local/cluster/shore:/local/cluster/SHOREmap:/local/cluster/BEAST/bin:/local/cluster/BEDTools/bin:/local/cluster/genomemapper:/local/cluster/iprscan/bin:/local/cluster/trinityrnaseq:/local/cluster/Cerulean/bin:/local/cluster/Quake/bin:/local/cluster/glimmer/bin:/local/cluster/samtools/bin:/local/cluster/SPAdes/bin:/local/cluster/RAPSearch2.16_64bits/bin:/local/cluster/last-418/bin:/local/cluster/rnammer:/local/cluster/SHRiMP/bin:/local/cluster/homer/bin:/local/cluster/cd-hit:/local/cluster/augustus/bin:/local/cluster/structure_linux_console/bin:/local/cluster/stampy:/local/cluster/infernal/binaries:/local/cluster/rtax:/local/cluster/pandaseq/bin:/local/cluster/bedtools2/bin:/local/cluster/htsjdk/dist:/local/cluster/GARM:/local/cluster/AmpliconNoise/ampliconnoise/Scripts:/local/cluster/AmpliconNoise/ampliconnoise:/local/cluster/pplacer-v1.1:/local/cluster/microbiomeutil/WigeoN:/local/cluster/microbiomeutil/TreeChopper:/local/cluster/microbiomeutil/NAST-iEr:/local/cluster/microbiomeutil/ChimeraSlayer:/local/cluster/AmosCmp16Spipeline:/local/cluster/Tisean_3.0.0/bin:/local/cluster/allpathslg/bin:/local/cluster/NAMD:/local/cluster/vcf/bin:/local/cluster/iRODS/clients/icommands/bin:/local/cluster/SVMerge/bin:/local/cluster/pindel/bin:/local/cluster/breakdancer-1.1.2/bin:/local/cluster/cnD/bin:/local/cluster/nextclip/bin:/local/cluster/prokka/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/julia-0.3.3/bin:/local/cluster/jnet/bin:/local/cluster/mongodb/bin:/local/cluster/gsl/bin:/local/cluster/sratoolkit/bin:/local/cluster/wise2.2.3-rc7/src/bin:/local/cluster/CEGMA_v2.5/bin:/local/cluster/freebayes/bin:/local/cluster/stacks/bin:/local/cluster/PhyloTreePruner:/local/cluster/barrnap-0.6/bin:/local/cluster/glpk/bin:/local/cluster/metaphlan/bin:/local/cluster/SOAPdenovo-Trans/bin:/local/cluster/ruby/bin:/local/cluster/fastqc:/local/cluster/PBSuite_15.8.24/bin:/local/cluster/detonate-1.10/bin:/local/cluster/canu/Linux-amd64/bin:/local/cluster/Blast2GO/bin:/local/cluster/JAGS-4.2.0/bin:/local/cluster/capnproto-c++/bin:/local/cluster/ucsc_genome:/local/cluster/vcflib/bin:/local/cluster/tre/bin:/local/cluster/MaSuRCA/bin:/local/cluster/BUSCO/bin:/local/cluster/BUSCO_v3.02/bin:/local/cluster/bib/active/bin:/local/cluster/Platypus/bin:/local/cluster/rMATS/bin:/local/cluster/sox/bin:/local/cluster/hisat2/bin:/local/cluster/bbmap:/local/cluster/mirdeep2/bin:/local/cluster/miniconda2/bin:/local/cluster/groff-1.22.3/bin:/local/cluster/EPACTS/bin:/local/cluster/percolator/bin:/local/cluster/mafft/bin:/local/cluster/gmap/bin:/local/cluster/BRAKER/bin:/local/cluster/MATLAB/R2018a/bin:/usr/X11R6/bin:/usr/X/bin:.:/local/cluster/dowser/bin:/local/cluster/dowser/bin/linux:/local/cluster/bamtools/bin:/nfs3/CGRB/home/gibbond/lib/edirect:/nfs3/CGRB/home/davised/opt/bin:/nfs3/CGRB/home/gibbond/local/bin/metaphlan2:/bin
cd /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair
+ cd /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.3.0
2019-07-18 10:48:28,091 - root - DEBUG - Running "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/task.json"
2019-07-18 10:48:28,094 - root - DEBUG - Checking existence of '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/task.json' with timeout=30
2019-07-18 10:48:28,096 - root - DEBUG - Loading JSON from '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/task.json'
2019-07-18 10:48:28,098 - root - DEBUG - {'bash_template_fn': 'template.sh',
 'inputs': {'AB_pairs': '../minced_bed/AB_pairs.txt',
            'minced_fasta': '../minced_fasta/minced.fasta'},
 'outputs': {'ov_index': 'ov_index.txt'},
 'parameters': {'pypeflow_mb': 4000, 'pypeflow_nproc': 8, 'topdir': '../..'}}
2019-07-18 10:48:28,098 - root - INFO - CD: '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair' <- '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair'
2019-07-18 10:48:28,099 - root - DEBUG - Checking existence of '../minced_bed/AB_pairs.txt' with timeout=30
2019-07-18 10:48:28,102 - root - DEBUG - Checking existence of '../minced_fasta/minced.fasta' with timeout=30
2019-07-18 10:48:28,105 - root - DEBUG - Checking existence of 'template.sh' with timeout=30
2019-07-18 10:48:28,109 - root - INFO - CD: '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair' <- '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair'
2019-07-18 10:48:28,116 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
fc_primary_contig_index.pl ../minced_bed/AB_pairs.txt ../minced_fasta/minced.fasta.fai > ov_index.txt
+ fc_primary_contig_index.pl ../minced_bed/AB_pairs.txt ../minced_fasta/minced.fasta.fai
perl: symbol lookup error: /usr/lib64/perl5/auto/Fcntl/Fcntl.so: undefined symbol: Perl_xs_apiversion_bootcheck
2019-07-18 10:48:28,227 - root - WARNING - Call '/bin/bash user_script.sh' returned 32512.
2019-07-18 10:48:28,227 - root - INFO - CD: '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair' -> '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair'
2019-07-18 10:48:28,228 - root - INFO - CD: '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair' -> '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair'
2019-07-18 10:48:28,229 - root - CRITICAL - Error in /nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 280, in <module>
    main()
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 272, in main
    run(**vars(parsed_args))
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 266, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 241, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash
    util.system(cmd)
  File "/nfs3/CGRB/home/gibbond/miniconda3/envs/bp-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 32512.
+++ pwd
++ echo 'FAILURE. Running top in /nfs3/CGRB/home/gibbond/projects/tyler-hi-c-phytophthora/test-data/F1_bull_test_data/5-phase/index-pair (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    0m1.083s
user    0m0.350s
sys     0m0.397s
+ finish
+ echo 'finish code: 1'

Thank you for the help!
-Dana

[pb-cdunn]

perl: symbol lookup error: /usr/lib64/perl5/auto/Fcntl/Fcntl.so: undefined symbol: Perl_xs_apiversion_bootcheck

That's disturbing. Your perl in Bioconda seems to be interacting with your system libraries. Not sure how to solve that. You could try conda update --all.

To debug, you can go into the run-directory of this failure (5-phase/index-pair/) and run the steps in the script yourself in your own shell. But it's unlikely we can solve this for you. If you hit a brick wall, maybe ask a question in a Bioconda forum.