Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cactus pipeline hangs when attempting to send TERMINATE signal to ktserver process under Singularity 3 #60

Open
lparsons opened this issue Feb 1, 2019 · 9 comments
Open

Cactus pipeline hangs when attempting to send TERMINATE signal to ktserver process under Singularity 3 #60

lparsons opened this issue Feb 1, 2019 · 9 comments

Comments

@lparsons
Copy link
Contributor

lparsons commented Feb 1, 2019 •
edited
Loading

When running the example data on our Centos6 system, interactively (no job manager) I run into a situation where the SavePrimaryDB step (among others) hangs. The problem is the DB service is not being terminated as it should be because the SIGINT signal isn't passed through the default Singularity 3 runscript. Under Singularity 2, the runscript used an exec command, but under Singularity 3, the runscript uses an eval command. This has the effect that signals are not passed down to the /opt/cactus/wrapper.sh script.

INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
WARNING:toil.resource:'JTRES_729ca706c404b9f57dd4bfd347bf0bd2' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
INFO:cactus.shared.common:Work dirs: set([])
INFO:cactus.shared.common:Docker work dir: .
INFO:cactus.shared.common:Running the command ['singularity', '--silent', 'run', '/Genomics/grid/users/lparsons/kocher_lab/cactus/cactus.img', 'cactus_workflow_flowerStats', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="128.112.116.245" port="9429" />\n\t\t</st_kv_database_conf>\n\t', '0']
Running command catchsegv 'cactus_workflow_flowerStats' 'INFO' '<st_kv_database_conf type=kyoto_tycoon> <kyoto_tycoon database_dir=fakepath host=128.112.116.245 port=9429 /> </st_kv_database_conf> ' '0'
Set log level to INFO
INFO:toil.fileStore:LOG-TO-MASTER: At end of caf phase, got stats {
  "flowerName": 0,
  "totalBases": 2254067,
  "totalEnds": 640,
  "totalCaps": 1642,
  "maxEndDegree": 4,
  "maxAdjacencyLength": 627671,
  "totalBlocks": 242,
  "totalGroups": 247,
  "totalEdges": 652,
  "totalFreeEnds": 146,
  "totalAttachedEnds": 10,
  "totalChains": 5,
  "totalLinkGroups": 247
}
flower name: 0 total bases: 2254067 total-ends: 640 total-caps: 1642 max-end-degree: 4 max-adjacency-length: 627671 total-blocks: 242 total-groups: 247 total-edges: 326 total-free-ends: 146 total-attached-ends: 10 total-chains: 5 total-link groups: 247

INFO:cactus.shared.common:Work dirs: set([])
INFO:cactus.shared.common:Docker work dir: .
INFO:cactus.shared.common:Running the command ['singularity', '--silent', 'run', '/Genomics/grid/users/lparsons/kocher_lab/cactus/cactus.img', 'ktremotemgr', 'set', '-port', '9429', '-host', '128.112.116.245', 'TERMINATE', '1']
Running command catchsegv 'ktremotemgr' 'set' '-port' '9429' '-host' '128.112.116.245' 'TERMINATE' '1'
@lparsons lparsons mentioned this issue Feb 1, 2019
Closed
@lparsons
Copy link
Contributor Author

lparsons commented Feb 1, 2019 •
edited
Loading

See my previous comments regarding this issue: #57 (comment), repeated here for clarity:

I am now simple trying to run the example workflow on our interactive node and it gets "stuck" on the SavePrimaryDB step every time. No errors that I can find, but things just stop. See attached logfile: cactus_localrun.log.gz

@lparsons
Copy link
Contributor Author

lparsons commented Feb 1, 2019

I'm presuming that the code is somehow stuck in this loop: https://github.com/ComparativeGenomicsToolkit/cactus/blob/master/src/cactus/pipeline/ktserverControl.py#L104

I can issue the command

singularity run /Genomics/grid/users/lparsons/kocher_lab/cactus/cactus.img ktremotemgr set -port 9429 -host 128.112.116.245 TERMINATE 1

and I get the response:

Running command catchsegv 'ktremotemgr' 'set' '-port' '9429' '-host' '128.112.116.245' 'TERMINATE' '1'

with no error, which makes me think there is no communication error, but that somehow the parameter isn't being recognized properly?

@lparsons
Copy link
Contributor Author

lparsons commented Feb 4, 2019

I'm fairly sure this issue is due to a problem sending the SIGINT signal to the ktserver process when running via singularity. I seem to get things moving again if I manually issue a SIGINT to the ktserver process.

cactus/src/cactus/pipeline/ktserverControl.py

Line 120 in d9039e7

process.send_signal(signal.SIGINT)

@lparsons lparsons changed the title Hang on SavePrimaryDB step in example data Cactus pipeline hangs when attempting to send TERMINATE signal to ktserver process under Singularity 3 Feb 6, 2019
@lparsons
Copy link
Contributor Author

lparsons commented Feb 6, 2019

I have identified a workaround for now, but would like to discuss a more robust solution going forward.

The workaround is to use a Singularity definition file to build the image, replacing the default runscript with one that uses exec. Then, use the new --containerImage option to specify this pre-built image. The container definition file (cactus_singularity_image.def) I used was:

Bootstrap: docker
From: quay.io/comparative-genomics-toolkit/cactus:latest

%runscript
    exec "bash" "/opt/cactus/wrapper.sh" "$@"

The command to build the image was:

sudo singularity build cactus.img cactus_singularity_image.def

@lparsons lparsons mentioned this issue Feb 6, 2019
Closed
@lparsons
Copy link
Contributor Author

This appears to be fixed in Singularity 3.1.0 release candidates.

lparsons added a commit to lparsons/cactus that referenced this issue Mar 12, 2019
@lparsons
Update README.md to note Singularity 3 issues
ddd42d3 
Fixes ComparativeGenomicsToolkit#55, Fixes ComparativeGenomicsToolkit#60
@lparsons lparsons mentioned this issue Mar 12, 2019
Closed
@ynanyam
Copy link

ynanyam commented Sep 20, 2019

I am using singularity 3.2.1 and still having the issue of ktserver process not being terminated.

I also tried your workaround of building a container with the definition file you suggested and the behaviour is the same.

@lparsons
Copy link
Contributor Author

Perhaps you could share some of the log output? It may be an unrelated issue?

@ynanyam
Copy link

ynanyam commented Sep 20, 2019

Sure.

INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:15:50: Before preprocessing, got assembly stats for genome af13: Input-sample: tmpiOpfrb.tmp Total-sequences: 8 Total-length: 37438924 Proportion-repeat-masked: 0.000043 ProportionNs: 0.000011 Total-Ns: 400 N50: 4650921 Median-sequence-length: 4650921 Max-sequence-length: 6783352 Min-sequence-length: 3015401

INFO:toil.leader:Job ended successfully: 'logAssemblyStats' v/T/job9jAVwH
INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 5/p/jobQidquO
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmphPg5W8/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpcVueeI/worker_log.txt
INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 1/S/jobASV9Y0
INFO:toil.leader:Issued job 'BatchPreprocessor' K/r/jobMiIgXf with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 9/w/jobg8JyOY
INFO:toil.leader:Issued job 'PreprocessSequence' U/b/job02eJl5 with job batch system ID: 16 and cores: 1, disk: 200.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpdzSc69/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' K/r/jobMiIgXf
INFO:toil.leader:Issued job 'PreprocessSequence' H/8/jobiELb9H with job batch system ID: 17 and cores: 1, disk: 200.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpmxdeEK/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp8SGeQQ/worker_log.txt
INFO:toil.leader:Job ended successfully: 'PreprocessSequence' U/b/job02eJl5
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' o/3/jobvfUeP5 with job batch system ID: 18 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Z/h/jobtcQQhv with job batch system ID: 19 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Z/S/jobWQzlSv with job batch system ID: 20 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' C/y/jobNxNLdS with job batch system ID: 21 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' X/U/jobF3nzgV with job batch system ID: 22 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' V/n/jobF0D5eO with job batch system ID: 23 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' d/E/jobta0ilb with job batch system ID: 24 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' W/A/job8kroiw with job batch system ID: 25 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' O/t/jobi4hVGh with job batch system ID: 26 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' o/L/job5eFiqQ with job batch system ID: 27 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' C/F/jobTPBHqo with job batch system ID: 28 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 8/z/jobQ_Mq23 with job batch system ID: 29 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Y/b/jobxeni3C with job batch system ID: 30 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Job ended successfully: 'PreprocessSequence' H/8/jobiELb9H
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' F/C/jobNYjByP with job batch system ID: 31 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' P/g/jobZElI03 with job batch system ID: 32 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' c/L/jobvPrMsP with job batch system ID: 33 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' y/p/job2e8fO0 with job batch system ID: 34 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' J/N/jobKxsJZ4 with job batch system ID: 35 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 5/b/job5aVhdK with job batch system ID: 36 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' x/Y/jobfbLN_U with job batch system ID: 37 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' k/x/jobnA0evN with job batch system ID: 38 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' r/i/jobRtCNgQ with job batch system ID: 39 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' l/H/jobMsQDVK with job batch system ID: 40 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' i/K/jobyRzLhQ with job batch system ID: 41 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 6/E/jobASnSoI with job batch system ID: 42 and cores: 1, disk: 100.0 M, and memory: 4.0 G
INFO:toil.leader:Issued job 'LastzRepeatMaskJob' G/Y/joblZPisM with job batch system ID: 43 and cores: 1, disk: 100.0 M, and memory: 4.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpDKfhKn/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpyYtSSo/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp4JVG_a/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpszzMwF/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpGkQ5LF/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpYsMhy7/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpSbfERJ/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpI7c0_K/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp7CPlyo/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpq9gFgK/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpCRXHzu/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpcCz38R/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpK_7tpm/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpXOzsur/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp9I6R2K/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmph_Gc9D/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpF4e5NI/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpjkK8oY/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpe9pwBT/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpxiSuVN/worker_log.txt
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmppZIG20/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpeLOSYE/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp0ECMRt/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmphh_VNJ/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp8l_tjw/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmppPH3qY/worker_log.txt
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Y/b/jobxeni3C
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' G/Y/joblZPisM
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 8/z/jobQ_Mq23
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 6/E/jobASnSoI
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' W/A/job8kroiw
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' k/x/jobnA0evN
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' O/t/jobi4hVGh
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' d/E/jobta0ilb
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Z/S/jobWQzlSv
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' o/L/job5eFiqQ
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' i/K/jobyRzLhQ
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' X/U/jobF3nzgV
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' P/g/jobZElI03
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' C/y/jobNxNLdS
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' c/L/jobvPrMsP
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' o/3/jobvfUeP5
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' V/n/jobF0D5eO
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' y/p/job2e8fO0
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' J/N/jobKxsJZ4
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 5/b/job5aVhdK
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' C/F/jobTPBHqo
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' x/Y/jobfbLN_U
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' l/H/jobMsQDVK
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' F/C/jobNYjByP
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Z/h/jobtcQQhv
INFO:toil.leader:Issued job 'MergeChunks' g/f/jobhfAOvb with job batch system ID: 44 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' r/i/jobRtCNgQ
INFO:toil.leader:Issued job 'MergeChunks' U/r/jobpxVmnB with job batch system ID: 45 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp8KTPR0/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MergeChunks' g/f/jobhfAOvb
INFO:toil.leader:Issued job 'MergeChunks2' v/s/jobAiIE7S with job batch system ID: 46 and cores: 1, disk: 100.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp_gFpWV/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MergeChunks' U/r/jobpxVmnB
INFO:toil.leader:Issued job 'MergeChunks2' M/h/jobez2nc9 with job batch system ID: 47 and cores: 1, disk: 100.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpt6cNb1/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp4nxzBO/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MergeChunks2' v/s/jobAiIE7S
INFO:toil.leader:Issued job 'MergeChunks' g/f/jobhfAOvb with job batch system ID: 48 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'MergeChunks2' M/h/jobez2nc9
INFO:toil.leader:Issued job 'MergeChunks' U/r/jobpxVmnB with job batch system ID: 49 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpwpgpgg/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MergeChunks' g/f/jobhfAOvb
INFO:toil.leader:Issued job 'PreprocessSequence' U/b/job02eJl5 with job batch system ID: 50 and cores: 1, disk: 200.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpIIB90p/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MergeChunks' U/r/jobpxVmnB
INFO:toil.leader:Issued job 'PreprocessSequence' H/8/jobiELb9H with job batch system ID: 51 and cores: 1, disk: 200.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmphtLHAs/worker_log.txt
INFO:toil.leader:Job ended successfully: 'PreprocessSequence' U/b/job02eJl5
INFO:toil.leader:Issued job 'BatchPreprocessor' 9/w/jobg8JyOY with job batch system ID: 52 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp3S_AYU/worker_log.txt
INFO:toil.leader:Job ended successfully: 'PreprocessSequence' H/8/jobiELb9H
INFO:toil.leader:Issued job 'BatchPreprocessor' K/r/jobMiIgXf with job batch system ID: 53 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpB1LT_m/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 9/w/jobg8JyOY
INFO:toil.leader:Issued job 'BatchPreprocessor' X/p/job_c5XFT with job batch system ID: 54 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpscUroR/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' K/r/jobMiIgXf
INFO:toil.leader:Issued job 'BatchPreprocessor' B/A/jobQFJYkN with job batch system ID: 55 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpf_wVuq/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' X/p/job_c5XFT
INFO:toil.leader:Issued job 'CactusPreprocessor2' F/s/job9VAFLt with job batch system ID: 56 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpq15kTt/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' B/A/jobQFJYkN
INFO:toil.leader:Issued job 'CactusPreprocessor2' x/G/jobkYYIxL with job batch system ID: 57 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpKmEYkJ/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpfOqpiG/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' F/s/job9VAFLt
INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' x/G/jobkYYIxL
INFO:toil.leader:Issued job 'CactusPreprocessor' F/E/jobq8ZKs9 with job batch system ID: 58 and cores: 1, disk: 100.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpxfuEdu/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' F/E/jobq8ZKs9
INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown2' 0/t/jobcu_nWj with job batch system ID: 59 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpxG9J5d/worker_log.txt
INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown2' 0/t/jobcu_nWj
INFO:toil.leader:Issued job 'logAssemblyStats' l/n/jobBYlLsd with job batch system ID: 60 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Issued job 'logAssemblyStats' p/p/jobsBZH6P with job batch system ID: 61 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Issued job 'ProgressiveDown' M/L/jobIrnqjB with job batch system ID: 62 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp3lqugI/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpGEpIvK/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpnRMQND/worker_log.txt
INFO:toil.leader:Job ended successfully: 'ProgressiveDown' M/L/jobIrnqjB
INFO:toil.leader:Issued job 'ProgressiveNext' j/X/jobsmH9hV with job batch system ID: 63 and cores: 1, disk: 2.0 G, and memory: 3.3 G
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:19: There are 0 dependent projects
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpYifu0y/worker_log.txt
INFO:toil.leader:Job ended successfully: 'logAssemblyStats' p/p/jobsBZH6P
INFO:toil.leader:Job ended successfully: 'logAssemblyStats' l/n/jobBYlLsd
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:20: Project has 0 dependencies
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:20: After preprocessing, got assembly stats for genome n3357: Input-sample: tmp8gdGNm.tmp Total-sequences: 8 Total-length: 36995582 Proportion-repeat-masked: 0.000692 ProportionNs: 0.000011 Total-Ns: 400 N50: 4658713 Median-sequence-length: 4658713 Max-sequence-length: 6386556 Min-sequence-length: 3033036

INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:20: After preprocessing, got assembly stats for genome af13: Input-sample: tmpDgsFv8.tmp Total-sequences: 8 Total-length: 37438924 Proportion-repeat-masked: 0.001028 ProportionNs: 0.000011 Total-Ns: 400 N50: 4650921 Median-sequence-length: 4650921 Max-sequence-length: 6783352 Min-sequence-length: 3015401

INFO:toil.leader:Job ended successfully: 'ProgressiveNext' j/X/jobsmH9hV
INFO:toil.leader:Issued job 'ProgressiveUp' H/E/jobbl6QcE with job batch system ID: 64 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpZpWGpF/worker_log.txt
INFO:toil.leader:Job ended successfully: 'ProgressiveUp' H/E/jobbl6QcE
INFO:toil.leader:Issued job 'CactusTrimmingBlastPhase' 2/L/jobBJ1JkD with job batch system ID: 65 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp3Zf6fk/worker_log.txt
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:23: Running blast using the trimming strategy
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:23: Ingroup sequences: [('af13', '2/L/jobBJ1JkD/g/tmpkv0Urq-run-tmp3zF6tN.tmp'), ('n3357', '2/L/jobBJ1JkD/g/tmpEUSSEm-run-tmpVx5oF7.tmp')]
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:23: Outgroup sequences: []
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:23: Running mapQ uniquifying with parameters, minimumMapQValue: 0.0, maxAlignmentsPerSite 5, alpha: 0.001
INFO:toil.leader:Job ended successfully: 'CactusTrimmingBlastPhase' 2/L/jobBJ1JkD
INFO:toil.leader:Issued job 'EncapsulatedJob' X/0/jobXzwWJQ with job batch system ID: 66 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpy4xzMw/worker_log.txt
INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' X/0/jobXzwWJQ
INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' x/1/jobLJTMzB with job batch system ID: 67 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpLlFKUm/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' x/1/jobLJTMzB
INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' w/b/joblqFroT with job batch system ID: 68 and cores: 1, disk: 300.0 M, and memory: 2.0 G
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:16:25: Blasting ingroups vs outgroups. Ingroup genomes: af13, n3357, outgroup genomes: 
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpNA7_g9/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' w/b/joblqFroT
INFO:toil.leader:Issued job 'MakeSelfBlasts' Y/P/jobpCncfi with job batch system ID: 69 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' a/c/job8DkzYg with job batch system ID: 70 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpfz9LpK/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp6dE2_T/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' Y/P/jobpCncfi
INFO:toil.leader:Issued job 'RunSelfBlast' f/g/jobdnoPue with job batch system ID: 71 and cores: 1, disk: 100.0 M, and memory: 100.0 M
INFO:toil.leader:Issued job 'RunSelfBlast' Y/V/joblM6oJ7 with job batch system ID: 72 and cores: 1, disk: 100.0 M, and memory: 100.0 M
INFO:toil.leader:Issued job 'RunSelfBlast' Z/C/job8PBOZl with job batch system ID: 73 and cores: 1, disk: 100.0 M, and memory: 100.0 M
INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' a/c/job8DkzYg
INFO:toil.leader:Issued job 'RunBlast' v/F/jobEqcUt0 with job batch system ID: 74 and cores: 1, disk: 100.0 M, and memory: 100.0 M
INFO:toil.leader:Issued job 'RunBlast' g/F/job9hwXMb with job batch system ID: 75 and cores: 1, disk: 100.0 M, and memory: 100.0 M
INFO:toil.leader:Issued job 'RunBlast' Q/k/jobCSBW_D with job batch system ID: 76 and cores: 1, disk: 100.0 M, and memory: 100.0 M
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmporLBy_/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpfdPoPE/worker_log.txt
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpkHxHFT/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmplONNCO/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmprBRjz0/worker_log.txt
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpVf5Gkl/worker_log.txt
INFO:toil.leader:Job ended successfully: 'RunSelfBlast' f/g/jobdnoPue
INFO:toil.leader:Job ended successfully: 'RunSelfBlast' Z/C/job8PBOZl
INFO:toil.leader:Job ended successfully: 'RunBlast' g/F/job9hwXMb
INFO:toil.leader:Job ended successfully: 'RunSelfBlast' Y/V/joblM6oJ7
INFO:toil.leader:Issued job 'CollateBlasts' i/R/job46RF69 with job batch system ID: 77 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp3Gwhhb/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' i/R/job46RF69
INFO:toil.leader:Issued job 'CollateBlasts2' d/q/jobHnKOh5 with job batch system ID: 78 and cores: 1, disk: 300.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpsUNMx2/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts2' d/q/jobHnKOh5
INFO:toil.leader:Issued job 'CollateBlasts' i/R/job46RF69 with job batch system ID: 79 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpOFBbwv/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' i/R/job46RF69
INFO:toil.leader:Issued job 'MakeSelfBlasts' Y/P/jobpCncfi with job batch system ID: 80 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpPBh9GD/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' Y/P/jobpCncfi
INFO:toil.leader:Job ended successfully: 'RunBlast' v/F/jobEqcUt0
INFO:toil.leader:Job ended successfully: 'RunBlast' Q/k/jobCSBW_D
INFO:toil.leader:Issued job 'CollateBlasts' J/n/jobJZQUaH with job batch system ID: 81 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpgPNJUa/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' J/n/jobJZQUaH
INFO:toil.leader:Issued job 'CollateBlasts2' a/Q/jobW4MS_f with job batch system ID: 82 and cores: 1, disk: 300.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmppSfhX4/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts2' a/Q/jobW4MS_f
INFO:toil.leader:Issued job 'CollateBlasts' J/n/jobJZQUaH with job batch system ID: 83 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpu9BqZx/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' J/n/jobJZQUaH
INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' a/c/job8DkzYg with job batch system ID: 84 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpdpDouP/worker_log.txt
INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' a/c/job8DkzYg
INFO:toil.leader:Issued job 'CollateBlasts' l/n/jobwKDJVr with job batch system ID: 85 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp4uUKWR/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' l/n/jobwKDJVr
INFO:toil.leader:Issued job 'CollateBlasts2' R/g/jobzBhEHq with job batch system ID: 86 and cores: 1, disk: 500.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp7iSA3Y/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts2' R/g/jobzBhEHq
INFO:toil.leader:Issued job 'CollateBlasts' l/n/jobwKDJVr with job batch system ID: 87 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpSgwyHJ/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CollateBlasts' l/n/jobwKDJVr
INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' w/b/joblqFroT with job batch system ID: 88 and cores: 1, disk: 300.0 M, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpzoGa1X/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' w/b/joblqFroT
INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' x/1/jobLJTMzB with job batch system ID: 89 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpv3seJG/worker_log.txt
INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' x/1/jobLJTMzB
INFO:toil.leader:Issued job 'Job' h/Z/jobAqGbw_ with job batch system ID: 90 and cores: 0, disk: 100.0 M, and memory: 512.0 M
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpZG59Zh/worker_log.txt
INFO:toil.leader:Job ended successfully: 'Job' h/Z/jobAqGbw_
INFO:toil.leader:Issued job 'mappingQualityRescoring' g/k/job4VQ9V4 with job batch system ID: 91 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpgtpBxb/worker_log.txt
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:33:18: Input cigar file has 269168 lines
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:33:18: Filtered, non-overlapping primary cigar file has 508268 lines
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:33:18: Filtered, non-overlapping secondary cigar file has 1893944 lines
INFO:toil.leader:Job ended successfully: 'mappingQualityRescoring' g/k/job4VQ9V4
INFO:toil.leader:Issued job 'Job' h/Z/jobAqGbw_ with job batch system ID: 92 and cores: 0, disk: 100.0 M, and memory: 512.0 M
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp4q_74X/worker_log.txt
INFO:toil.leader:Job ended successfully: 'Job' h/Z/jobAqGbw_
INFO:toil.leader:Issued job 'EncapsulatedJob' X/0/jobXzwWJQ with job batch system ID: 93 and cores: 1, disk: 2.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpTg7hzZ/worker_log.txt
INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' X/0/jobXzwWJQ
INFO:toil.leader:Issued job 'CactusSetupCheckpoint' S/T/jobkhxrmq with job batch system ID: 94 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpQOiU0l/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusSetupCheckpoint' S/T/jobkhxrmq
INFO:toil.leader:Issued job 'StartPrimaryDB' 8/U/jobKYyMqR with job batch system ID: 95 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpdxuoXn/worker_log.txt
INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' 8/U/jobKYyMqR
INFO:toil.leader:Issued job 'KtServerService' H/c/jobSYg6Fa with job batch system ID: 96 and cores: 0, disk: 2.0 G, and memory: 2.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp9UVRwc/worker_log.txt
INFO:toil.leader:Issued job 'CactusSetupPhase' 6/e/jobw3n0LT with job batch system ID: 97 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpAsMOfo/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 6/e/jobw3n0LT
INFO:toil.leader:Issued job 'CactusCafPhase' 7/D/jobIhRbPS with job batch system ID: 98 and cores: 1, disk: 2.0 G, and memory: 700.0 M
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpy6qfcy/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusCafPhase' 7/D/jobIhRbPS
INFO:toil.leader:Issued job 'CactusCafWrapper' i/W/jobh1Z4jj with job batch system ID: 99 and cores: 1, disk: 2.0 G, and memory: 1.4 G
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:33:47: Converted headers of cigar file G/T/tmpC_3rhp-mappingQualityRescoring-tmp6ubH6I.tmp to internal names, new file /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpy6qfcy/bf43d0bf-53cf-4fd0-ad5e-2f8bb0353bf2/tmpilsys0.tmp
INFO:toil.statsAndLogging:Got message from job at time 09-20-2019 11:33:47: Converted headers of secondary cigar file q/A/tmpsr2kJ9-mappingQualityRescoring-tmpPBg8An.tmp to internal names, new file /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpy6qfcy/bf43d0bf-53cf-4fd0-ad5e-2f8bb0353bf2/tmpBwvuLU.tmp
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmp3Z0c2G/worker_log.txt
INFO:toil.leader:Job ended successfully: 'CactusCafWrapper' i/W/jobh1Z4jj
INFO:toil.leader:Issued job 'SavePrimaryDB' D/C/job8vcwW6 with job batch system ID: 100 and cores: 1, disk: 2.0 G, and memory: 3.3 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/local/bgfs/yasasvy.nanyam/948085/jobStore'.
INFO:toil.worker:Redirecting logging to /local/bgfs/yasasvy.nanyam/948085/toil-b4f0c6a3-b008-4775-bd35-50319cd5fee9-92eafaf6-d777-4a90-84a8-dc876ae23e77/tmpY8XB6_/worker_log.txt

And from the log of the particular job

INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
WARNING:toil.resource:'JTRES_54ac0ad9d8793c4f87b983ea2a8ba591' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
INFO:cactus.shared.common:Work dirs: set([])
INFO:cactus.shared.common:Docker work dir: .
INFO:cactus.shared.common:Running the command ['singularity', '--silent', 'run', u'/local/bgfs/yasasvy.nanyam/948085/jobStore/cactus.img', 'cactus_workflow_flowerStats', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="<ip address>" port="30627" />\n\t\t</st_kv_database_conf>\n\t', '0']
Running command catchsegv 'cactus_workflow_flowerStats' 'INFO' '<st_kv_database_conf type=kyoto_tycoon>
			<kyoto_tycoon database_dir=fakepath host=<ip address> port=30627 />
		</st_kv_database_conf>
	' '0'
Set log level to INFO
INFO:toil.fileStore:LOG-TO-MASTER: At end of caf phase, got stats {
  "flowerName": 0,
  "totalBases": 74434506,
  "totalEnds": 33748,
  "totalCaps": 72936,
  "maxEndDegree": 9,
  "maxAdjacencyLength": 53603,
  "totalBlocks": 16858,
  "totalGroups": 16426,
  "totalEdges": 34684,
  "totalFreeEnds": 16,
  "totalAttachedEnds": 16,
  "totalChains": 630,
  "totalLinkGroups": 16186
}
flower name: 0 total bases: 74434506 total-ends: 33748 total-caps: 72936 max-end-degree: 9 max-adjacency-length: 53603 total-blocks: 16858 total-groups: 16426 total-edges: 17342 total-free-ends: 16 total-attached-ends: 16 total-chains: 630 total-link groups: 16186

INFO:cactus.shared.common:Work dirs: set([])
INFO:cactus.shared.common:Docker work dir: .
INFO:cactus.shared.common:Running the command ['singularity', '--silent', 'run', u'/local/bgfs/yasasvy.nanyam/948085/jobStore/cactus.img', 'ktremotemgr', 'set', '-port', '30627', '-host', '<ip address>', 'TERMINATE', '1']
Running command catchsegv 'ktremotemgr' 'set' '-port' '30627' '-host' '<ip address>' 'TERMINATE' '1'

General version info

[yasasvy.nanyam@sn-cn-19-1 cactus]$ singularity --version
singularity version 3.2.1-1.el7
[yasasvy.nanyam@sn-cn-19-1 cactus]$ uname -r
3.10.0-957.21.2.el7.x86_64
[yasasvy.nanyam@sn-cn-19-1 cactus]$ cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

@legfed-build
Copy link

A colleague and I discovered (in the context of a different application) that Singularity 3.2 wasn't propagating signals to processes within the container. We Slacked the Singularity developers, and the issue was fixed in Singularity 3.3 (sylabs/singularity@f7b429f)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update README.md to note Singularity 3 issues lparsons/cactus
3 participants
@lparsons @ynanyam @legfed-build