[ERROR]Task Node(0-rawreads/build) failed with exit-code=1 using SLURM #109
Labels
Comments
|
This was originally PacificBiosciences/FALCON#685 I wrote:
You've got to look into |
|
Here is the actual error, from near the bottom of your The current problem is that you are providing |
|
Thanks, Chris!
…On Thu, Mar 14, 2019 at 9:44 AM Christopher Dunn ***@***.***> wrote:
Closed #109 <#109>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#109 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AUPt76mXkrDDYsd6IzUUAIMcJhwiN7MXks5vWnx1gaJpZM4buAy_>
.
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu <kiv@berkeley.edu>
|
|
Hi Chris,
This run went a lot longer, but returned another error. I can't seem to
find any usage options suggesting I used the wrong commands anywhere else:
```
executable=${PYPEFLOW_JOB_START_SCRIPT}
+
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60
# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done
+ [[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
+
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60
# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done
+ [[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
+
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60
# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done
+ [[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]
/bin/bash ${executable}
+ /bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
/bin/bash ${executable}
+ /bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
/bin/bash ${executable}
+ /bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
executable=${PYPEFLOW_JOB_START_SCRIPT}
+
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60
# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done
+ [[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]
/bin/bash ${executable}
+ /bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
executable=${PYPEFLOW_JOB_START_SCRIPT}
+
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60
# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done
+ [[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]
/bin/bash ${executable}
+ /bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
+ '[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
+ '[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
+ '[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
+ '[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
+ '[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
+
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
+ /bin/bash task.sh
export PATH=$PATH:/bin
+ export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
+
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
+ /bin/bash task.sh
export PATH=$PATH:/bin
+ export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
+
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
+ /bin/bash task.sh
export PATH=$PATH:/bin
+ export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
+
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
+ /bin/bash task.sh
export PATH=$PATH:/bin
+ export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
+
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
+ cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,261 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,261 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,262 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,262 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,263 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,265 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,265 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,600 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
hostname
+ hostname
hostname
+ hostname
hostname
+ hostname
pwd
+ pwd
date
+ date
pwd
+ pwd
date
+ date
hostname
+ hostname
pwd
+ pwd
date
+ date
pwd
+ pwd
date
+ date
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
+ python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
+ python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
+ python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
+ python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
+ python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480 -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480 -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480 -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480 -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480 -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480 -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480 -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480 -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480 -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480 -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480 -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480 -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480 -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480 -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480 -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480 -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480 -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480 -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480 -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480 -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
module () { eval `/usr/bin/modulecmd bash $*`
}
module () { eval `/usr/bin/modulecmd bash $*`
}
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () { eval `/usr/bin/modulecmd bash $*`
}
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () { eval `/usr/bin/modulecmd bash $*`
}
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
module () { eval `/usr/bin/modulecmd bash $*`
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
}
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
+ rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
+ rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
+ rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
+ rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
+ rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
+ zmw_whitelist_option=
use_subsampling=0
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
+ use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed
"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
+ zmw_whitelist_option=
use_subsampling=0
+ use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed
"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
+ [[ 0 -eq 1 ]]
while read fn; do cat ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
+ [[ 0 -eq 1 ]]
+ zmw_whitelist_option=
use_subsampling=0
+ use_subsampling=0
while read fn; do cat ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed
"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
+ [[ 0 -eq 1 ]]
while read fn; do cat ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
+ read fn
+ cat ../../m54050R1_180201_210150.subreads.fasta
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
+ zmw_whitelist_option=
use_subsampling=0
+ use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed
"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
+ [[ 0 -eq 1 ]]
while read fn; do cat ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
+ read fn
+ read fn
+ read fn
+ fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ cat ../../m54050R1_180201_210150.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
+ zmw_whitelist_option=
use_subsampling=0
+ use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed
"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
+ [[ 0 -eq 1 ]]
while read fn; do cat ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
+ read fn
+ cat ../../m54050R1_180201_210150.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
+ cat ../../m54050R1_180201_210150.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ cat ../../m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 224
INFO:root: #2 count= 839
INFO:root: #3 count= 2,102
INFO:root: #4 count= 4,170
INFO:root: #6 count= 8,348
INFO:root: #9 count= 14,686
INFO:root: #11 count= 28,274
INFO:root: #113 count= 48,575
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 224
INFO:root: #2 count= 839
INFO:root: #3 count= 2,102
INFO:root: #4 count= 4,170
INFO:root: #6 count= 8,348
INFO:root: #9 count= 14,686
INFO:root: #11 count= 28,274
INFO:root: #113 count= 48,575
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 224
INFO:root: #2 count= 839
INFO:root: #3 count= 2,102
INFO:root: #4 count= 4,170
INFO:root: #6 count= 8,348
INFO:root: #9 count= 14,686
INFO:root: #11 count= 28,274
INFO:root: #113 count= 48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 224
INFO:root: #2 count= 839
INFO:root: #3 count= 2,102
INFO:root: #4 count= 4,170
INFO:root: #6 count= 8,348
INFO:root: #9 count= 14,686
INFO:root: #11 count= 28,274
INFO:root: #113 count= 48,575
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 224
INFO:root: #2 count= 839
INFO:root: #3 count= 2,102
INFO:root: #4 count= 4,170
INFO:root: #6 count= 8,348
INFO:root: #9 count= 14,686
INFO:root: #11 count= 28,274
INFO:root: #113 count= 48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root: #317 count= 89,000
INFO:root: #317 count= 89,000
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root: #317 count= 89,000
INFO:root: #317 count= 89,000
INFO:root: #317 count= 89,000
INFO:root: #492 count= 178,009
INFO:root: #492 count= 178,009
INFO:root: #492 count= 178,009
INFO:root: #492 count= 178,009
INFO:root: #492 count= 178,009
INFO:root: #507 count= 355,936
INFO:root: #507 count= 355,936
INFO:root: #507 count= 355,936
INFO:root: #507 count= 355,936
INFO:root: #507 count= 355,936
INFO:root: #612 count= 692,452
INFO:root: #612 count= 692,452
INFO:root: #612 count= 692,452
INFO:root: #612 count= 692,452
INFO:root: #612 count= 692,452
INFO:root: #687 count= 1,370,718
INFO:root: #687 count= 1,370,718
INFO:root: #687 count= 1,370,718
INFO:root: #687 count= 1,370,718
INFO:root: #687 count= 1,370,718
INFO:root: #816 count= 2,688,290
INFO:root: #816 count= 2,688,290
INFO:root: #816 count= 2,688,290
INFO:root: #816 count= 2,688,290
INFO:root: #816 count= 2,688,290
INFO:root: #1,375 count= 5,277,506
INFO:root: #1,375 count= 5,277,506
INFO:root: #1,375 count= 5,277,506
INFO:root: #1,375 count= 5,277,506
INFO:root: #1,375 count= 5,277,506
INFO:root: #2,367 count= 10,451,057
INFO:root: #2,367 count= 10,451,057
INFO:root: #2,367 count= 10,451,057
INFO:root: #2,367 count= 10,451,057
INFO:root: #2,367 count= 10,451,057
INFO:root: #4,437 count= 20,814,122
INFO:root: #4,437 count= 20,814,122
INFO:root: #4,437 count= 20,814,122
INFO:root: #4,437 count= 20,814,122
INFO:root: #4,437 count= 20,814,122
INFO:root: #9,932 count= 41,507,957
INFO:root: #9,932 count= 41,507,957
INFO:root: #9,932 count= 41,507,957
INFO:root: #9,932 count= 41,507,957
INFO:root: #9,932 count= 41,507,957
INFO:root: #17,666 count= 82,897,567
INFO:root: #17,666 count= 82,897,567
INFO:root: #17,666 count= 82,897,567
INFO:root: #17,666 count= 82,897,567
INFO:root: #17,666 count= 82,897,567
INFO:root: #32,172 count= 165,669,678
INFO:root: #32,172 count= 165,669,678
INFO:root: #32,172 count= 165,669,678
INFO:root: #32,172 count= 165,669,678
INFO:root: #32,172 count= 165,669,678
INFO:root: #58,663 count= 331,230,374
INFO:root: #58,663 count= 331,230,374
INFO:root: #58,663 count= 331,230,374
INFO:root: #58,663 count= 331,230,374
INFO:root: #58,663 count= 331,230,374
INFO:root: #111,773 count= 662,326,916
INFO:root: #111,773 count= 662,326,916
INFO:root: #111,773 count= 662,326,916
INFO:root: #111,773 count= 662,326,916
INFO:root: #111,773 count= 662,326,916
INFO:root: #228,302 count= 1,324,520,840
INFO:root: #228,302 count= 1,324,520,840
INFO:root: #228,302 count= 1,324,520,840
INFO:root: #228,302 count= 1,324,520,840
INFO:root: #228,302 count= 1,324,520,840
INFO:root: #456,327 count= 2,648,877,672
INFO:root: #456,327 count= 2,648,877,672
INFO:root: #456,327 count= 2,648,877,672
INFO:root: #456,327 count= 2,648,877,672
INFO:root: #456,327 count= 2,648,877,672
+ read fn
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ cat ../../m54050R1_180205_001444.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
+ read fn
+ read fn
+ read fn
+ cat ../../m54050R1_180205_001444.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
+ cat ../../m54050R1_180205_001444.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
+ cat ../../m54050R1_180205_001444.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ read fn
+ cat ../../m54050R1_180205_001444.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 14,054
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 14,054
INFO:root: #3 count= 44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 14,054
INFO:root: #3 count= 44,205
INFO:root: #3 count= 44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root: #5 count= 99,813
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root: #5 count= 99,813
INFO:root: #6 count= 164,238
INFO:root: #5 count= 99,813
INFO:root: #6 count= 164,238
INFO:root:Counting bytes from "None"
INFO:root: #6 count= 164,238
INFO:root: #1 count= 14,054
INFO:root: #3 count= 44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root: #5 count= 99,813
INFO:root: #6 count= 164,238
INFO:root: #12 count= 285,279
INFO:root: #12 count= 285,279
INFO:root: #12 count= 285,279
INFO:root: #12 count= 285,279
INFO:root: #36 count= 538,708
INFO:root: #36 count= 538,708
pypeflow 2.1.1
INFO:root: #36 count= 538,708
INFO:root: #36 count= 538,708
INFO:root: #128 count= 996,561
INFO:root: #128 count= 996,561
INFO:root: #128 count= 996,561
INFO:root: #128 count= 996,561
INFO:root: #233 count= 1,916,542
INFO:root: #233 count= 1,916,542
INFO:root: #233 count= 1,916,542
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 14,054
INFO:root: #3 count= 44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root: #5 count= 99,813
INFO:root: #233 count= 1,916,542
INFO:root: #6 count= 164,238
INFO:root: #12 count= 285,279
INFO:root: #36 count= 538,708
INFO:root: #128 count= 996,561
INFO:root: #359 count= 3,717,003
INFO:root: #359 count= 3,717,003
INFO:root: #359 count= 3,717,003
INFO:root: #359 count= 3,717,003
INFO:root: #233 count= 1,916,542
INFO:root: #359 count= 3,717,003
INFO:root: #1,096 count= 7,315,096
INFO:root: #1,096 count= 7,315,096
INFO:root: #1,096 count= 7,315,096
INFO:root: #1,096 count= 7,315,096
INFO:root: #1,096 count= 7,315,096
INFO:root: #2,198 count= 14,516,279
INFO:root: #2,198 count= 14,516,279
INFO:root: #2,198 count= 14,516,279
INFO:root: #2,198 count= 14,516,279
INFO:root: #2,198 count= 14,516,279
INFO:root: #5,484 count= 28,909,411
INFO:root: #5,484 count= 28,909,411
INFO:root: #5,484 count= 28,909,411
INFO:root: #5,484 count= 28,909,411
INFO:root: #5,484 count= 28,909,411
INFO:root: #10,469 count= 57,692,955
INFO:root: #10,469 count= 57,692,955
INFO:root: #10,469 count= 57,692,955
INFO:root: #10,469 count= 57,692,955
INFO:root: #10,469 count= 57,692,955
INFO:root: #18,997 count= 115,259,072
INFO:root: #18,997 count= 115,259,072
INFO:root: #18,997 count= 115,259,072
INFO:root: #18,997 count= 115,259,072
INFO:root: #18,997 count= 115,259,072
INFO:root: #37,811 count= 230,392,828
INFO:root: #37,811 count= 230,392,828
INFO:root: #37,811 count= 230,392,828
INFO:root: #37,811 count= 230,392,828
INFO:root: #37,811 count= 230,392,828
INFO:root: #71,452 count= 460,654,500
INFO:root: #71,452 count= 460,654,500
INFO:root: #71,452 count= 460,654,500
INFO:root: #71,452 count= 460,654,500
INFO:root: #71,452 count= 460,654,500
INFO:root: #143,311 count= 921,176,072
INFO:root: #143,311 count= 921,176,072
INFO:root: #143,311 count= 921,176,072
INFO:root: #143,311 count= 921,176,072
INFO:root: #143,311 count= 921,176,072
INFO:root: #301,804 count= 1,842,225,458
INFO:root: #301,804 count= 1,842,225,458
INFO:root: #301,804 count= 1,842,225,458
INFO:root: #301,804 count= 1,842,225,458
INFO:root: #301,804 count= 1,842,225,458
INFO:root: #632,056 count= 3,684,320,381
INFO:root: #632,056 count= 3,684,320,381
INFO:root: #632,056 count= 3,684,320,381
INFO:root: #632,056 count= 3,684,320,381
INFO:root: #632,056 count= 3,684,320,381
+ read fn
+ read fn
+ cat ../../m54050R1_180205_102507.subreads.fasta
+ cat ../../m54050R1_180205_102507.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
+ read fn
+ cat ../../m54050R1_180205_102507.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ read fn
+ fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
+ cat ../../m54050R1_180205_102507.subreads.fasta
+ fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ read fn
+ cat ../../m54050R1_180205_102507.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter streamed-median -
+ fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 201
INFO:root:Counting bytes from "None"
INFO:root: #2 count= 422
INFO:root: #4 count= 884
INFO:root: #8 count= 1,773
INFO:root: #15 count= 3,383
INFO:root: #1 count= 201
INFO:root: #2 count= 422
INFO:root: #4 count= 884
INFO:root: #8 count= 1,773
INFO:root: #15 count= 3,383
INFO:root: #29 count= 6,612
INFO:root: #57 count= 13,179
INFO:root: #61 count= 37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
pypeflow 2.1.1
INFO:root: #29 count= 6,612
INFO:root: #57 count= 13,179
INFO:root: #61 count= 37,191
Adding 'm54050R1_180205_102507.subreads.fasta.fasta' ...
INFO:root: #63 count= 76,347
INFO:root: #71 count= 134,764
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 201
INFO:root: #2 count= 422
INFO:root: #4 count= 884
INFO:root: #8 count= 1,773
INFO:root: #15 count= 3,383
INFO:root: #29 count= 6,612
INFO:root: #57 count= 13,179
INFO:root: #61 count= 37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
INFO:root: #425 count= 243,370
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
"__main__", fname, loader, pkg_name)
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in <module>
exec code in run_globals
main(sys.argv) # pragma: no cover
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in <module>
main(sys.argv) # pragma: no cover
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
IOError: [Errno 32] Broken pipe
INFO:root: #442 count= 449,237
cat: write error: Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
[WARNING]Call 'bash -vex build_db.sh' returned 256.
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in <module>
main()
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in <module>
main()
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
args.func(args)
INFO:root:Counting bytes from "None"
INFO:root: #1 count= 201
INFO:root: #2 count= 422
INFO:root: #4 count= 884
INFO:root: #8 count= 1,773
args.func(args)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py",
line 29, in syscall
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
INFO:root: #15 count= 3,383
INFO:root: #29 count= 6,612
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
INFO:root: #57 count= 13,179
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 167, in build_db
INFO:root: #61 count= 37,191
io.syscall('bash -vex {}'.format(script_fn))
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py",
line 29, in syscall
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in <module>
INFO:root:Counting bytes from "None"
main(sys.argv) # pragma: no cover
INFO:root: #1 count= 201
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
INFO:root: #2 count= 422
args.func(args)
INFO:root: #4 count= 884
INFO:root: #8 count= 1,773
INFO:root: #657 count= 861,953
INFO:root: #15 count= 3,383
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
INFO:root: #29 count= 6,612
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
INFO:root: #57 count= 13,179
INFO:root: #61 count= 37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in <module>
main(sys.argv) # pragma: no cover
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in <module>
INFO:root: #890 count= 1,685,294
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in <module>
main()
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
args.func(args)
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File
"/pylon5/mc5fqip/orca21/
```
|
Hi, I checked the stderr file in '0-rawreads/build', and found the following error message: How can I fix this ? Thanks THIS IS THE DFG FILE: Input[General]
input_fofn=/BIGDATA1/wbgcas_tfeng_1/projcet/pb_assembly_test/input.fofn
input_type=raw
pa_DBdust_option=
pa_fasta_filter_option=pass
target=assembly
skip_checks=False
LA4Falcon_preload=false
#### Data Partitioning
pa_DBsplit_option=-x500 -s200
ovlp_DBsplit_option=-x500 -s200
#### Repeat Masking
pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300
####Pre-assembly
genome_size=0
seed_coverage=20
length_cutoff=1000
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e.8 -l2000 -k18 -h480 -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 1800
falcon_sense_greedy=False
####Pread overlapping
ovlp_daligner_option=-e.9 -l2500 -k24 -h1024 -w6 -s100
ovlp_HPCdaligner_option=-v -B128 -M24
####Final Assembly
overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=
length_cutoff_pr=1000
[job.defaults]
job_type=slurm
pwatcher_type=blocking
JOB_QUEUE = free
JOB_NAME = 200kb_test
MB = 32768
NPROC = 6
njobs = 32
submit = srun --wait=0 -p ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} --mem-per-cpu=${MB}M --cpus-per-task=${NPROC} ${JOB_SCRIPT}
[job.step.da]
NPROC=4
MB=32768
njobs=240
[job.step.la]
NPROC=4
MB=32768
njobs=240
[job.step.cns]
NPROC=8
MB=65536
njobs=240
[job.step.pda]
NPROC=4
MB=32768
njobs=240
[job.step.pla]
NPROC=4
MB=32768
njobs=240
[job.step.asm]
NPROC=24
MB=64608
njobs=1 |
|
I copied your message into WordPad and found this: There are a variety of ways that can happen, e.g. you could have run out of disk-space, or something could have killed one of the processes. You'll need to debug this at your end. Go into the failing task directory ( |
|
Hi Chris,
I was able to narrow it down to this error:
#cat fc.fofn | xargs rm -f
DBdust raw_reads
+ DBdust raw_reads
Message from root@br006.pvt.bridges.psc.edu on <no tty> at 15:14 ...
pid 17944 (DBdust) killed: rss memory 4194.66 MB exceeded limit 4096 MB
Computations using more than 1800 cpu seconds
and/or 4096 MB of memory should use the batch system.
I have the following batch configuration:
#SBATCH -t 336:00:00
#SBATCH -N 1
#SBATCH --mem=1TB
#SBATCH --mem-per-cpu=5000M
#SBATCH --cpus-per-task=4
Nicolas
…On Sat, Mar 16, 2019 at 7:32 AM Christopher Dunn ***@***.***> wrote:
Reopened #109
<#109>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#109 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AUPt7-2ukDp1iDfBg3m4BvDGX36tBnwBks5vXQBugaJpZM4buAy_>
.
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu <kiv@berkeley.edu>
|
|
Yep, you ran out of memory. And DBdust is single-threaded/non-parallel. (So using more machines would not help.) You can try different options to |
|
With the following Config file: Input for FALCON 3/18/19[General] Data Partitioningpa_DBsplit_option=-x500 -s50 Repeat Maskingpa_HPCTANmask_option= ####Pre-assembly ####Pread overlapping ####Final Assembly [job.defaults] And the following script: #!/bin/bash #SBATCH -p LM source activate myenv run it!fc_run fc_run.cfg I have the following new stderr output: Any help is appreciated. executable=${PYPEFLOW_JOB_START_SCRIPT}
Wait up to timeout seconds for the executable to become "executable",then exec.#timeleft = int(timeout)
Wait up to timeout seconds for the executable to become "executable",then exec.#timeleft = int(timeout)
Wait up to timeout seconds for the executable to become "executable",then exec.#timeleft = int(timeout)
/bin/bash ${executable}
/bin/bash ${executable}
/bin/bash ${executable}
Wait up to timeout seconds for the executable to become "executable",then exec.#timeleft = int(timeout)
/bin/bash ${executable}
Wait up to timeout seconds for the executable to become "executable",then exec.#timeleft = int(timeout)
/bin/bash ${executable}
Substitution will be similar to snakemake "shell".python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
Substitution will be similar to snakemake "shell".python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
Substitution will be similar to snakemake "shell".python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
Substitution will be similar to snakemake "shell".python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
Substitution will be similar to snakemake "shell".python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
real 4m44.057s real 4m44.060s
real 4m44.080s
real 4m44.091s
DBsplit -f -a -x500 -s200 raw_reads
CUTOFF=$(python2.7 -m falcon_kit.mains.calc_cutoff --coverage 30.0 1100000000 <(DBstats -b1 raw_reads))
TODO: Verify that db exists.#ln -sf length_cutoff length_cutoff date
real 83m38.191s
|
We interact with GitHub so we can learn about small bugs directly from users, but this is a free service. Management does not give us much bandwidth to respond to GitHub tickets. Please do everything you can to reduce the time we need to spend. If you prove that we have a bug, we will try to fix it quickly. If you are running Sequel and using official PB "smrtlink" software, you can get more support through your PacBio representatives. |
|
Hi Chris,
The errors seemed different to me and I’m new to bioinformatics. My
apologies, it won’t happen again. I’ll keep these tips in mind.
Nicolas
Thank you for the info this is helpful.
On Tue, Mar 26, 2019 at 11:53 AM Christopher Dunn ***@***.***> wrote:
1. Did you cross-post this? It looks a lot like
PacificBiosciences/FALCON#588
<PacificBiosciences/FALCON#588>
2. If you're going to use GitHub, you need to learn to use GitHub. You
can "attach" a file, or you can post it somewhere (e.g. gist.github.com),
or you can post it here *with markdown formatting* if it's small.
Other users are able to figure this out. You're just dumping a lot of text.
3. You need to debug this COMPLETELY OUTSIDE THE WORKFLOW. (See my
response to the cross-post.) The bug appears to be specific to your
hardware. You should be able to run any task yourself, independent of the
workflow, and to learn how much memory it consumes.
We interact with GitHub so we can learn about small bugs directly from
users, but this is a free service. Management does not give us much
bandwidth to respond to GitHub tickets. Please do everything you can to
reduce the time we need to spend. If you prove that we have a bug, we will
try to fix it quickly.
If you are running Sequel and using official PB "smrtlink" software, you
can get more support through your PacBio representatives.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#109 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AUPt78obhTkmETmUFsfXkDuo-9WN1LLzks5vamyhgaJpZM4buAy_>
.
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu <kiv@berkeley.edu>
|
|
Hi nicolasalexandre21, Have you solved your problems? I got similar issues with slurm, but sloved by addiing the options : Let me know whether this helps. |
|
Thanks. I've updated the pypeflow wiki with that helpful advice. |
|
Hi Xiaoyezao,
Thank you for the helpful advice!! It seems to have mitigated the number of
error messages significantly. I am still getting a couple minor internal
allocation errors, but I will give you an update once I figure out the
problem in case it is helpful to anybody.
Nicolas
…On Fri, Mar 29, 2019 at 7:05 AM Christopher Dunn ***@***.***> wrote:
Thanks. I've updated the pypeflow wiki with that helpful advice.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#109 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AUPt74L3XLvFiGhHQ2g0rRUL3owBYVIPks5vbh2dgaJpZM4buAy_>
.
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu <kiv@berkeley.edu>
|
|
Hi Chris and Xiaoyezao,
It is finally running. We were able to get it to run by adding ${CMD} in
the srun command since falcon constructs values for ${CMD} that are then
launched by srun.
Thank you both for your help,
Nicolas
On Fri, Mar 29, 2019 at 9:20 AM Nicolas Alexandre <nalexandre@berkeley.edu>
wrote:
… Hi Xiaoyezao,
Thank you for the helpful advice!! It seems to have mitigated the number
of error messages significantly. I am still getting a couple minor internal
allocation errors, but I will give you an update once I figure out the
problem in case it is helpful to anybody.
Nicolas
On Fri, Mar 29, 2019 at 7:05 AM Christopher Dunn ***@***.***>
wrote:
> Thanks. I've updated the pypeflow wiki with that helpful advice.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#109 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AUPt74L3XLvFiGhHQ2g0rRUL3owBYVIPks5vbh2dgaJpZM4buAy_>
> .
>
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
***@***.*** ***@***.***>
--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu <kiv@berkeley.edu>
|
|
I should have mentioned: Always start with a tiny testcase. Integration problems are the worst! Glad you got it working. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Job script:
Config file
Error
Conda list
The text was updated successfully, but these errors were encountered: