You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
$namespaces:
cwltool: "http://commonwl.org/cwltool#"doc: |
Trivial MPI test that prints the process IDs of each of the parallel
processes. Requires Python (but you have cwltool running, right?)
and an MPI implementation.
baseCommand: python
requirements:
cwltool:MPIRequirement:
processes: 2
arguments: [-c, 'import os; print(os.getpid())']
inputs: []
outputs:
pids:
type: stdout
MPIRequirement is part of the cwltool extensions. This workflow also works with both cwltool and toil-cwl-runner given the right flags, ex:
$ export PATH=$PATH:/home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/ # where fake_mpi_run.py exists
$ cwltool --enable-dev --enable-ext --mpi-config-file /home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/fake_mpi.yml file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
INFO /home/heaucques/Documents/toil/venv3.12/bin/cwltool 3.1.20240508115724
INFO [job mpi_simple.cwl] /tmp/1ez0it5c$ fake_mpi_run.py \
--num \
2 \
--no-fail \
python \
-c \
'import os; print(os.getpid())' > /tmp/1ez0it5c/c83195d490fa84f03f0cb9b142c6f385d736bd94
INFO [job mpi_simple.cwl] completed success
{
"pids": {
"location": "file:///home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94",
"basename": "c83195d490fa84f03f0cb9b142c6f385d736bd94",
"class": "File",
"checksum": "sha1$010210c4a6298629ae0653165e70c393bc831cad",
"size": 12,
"path": "/home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94"
}
}INFO Final process status is success
However, running cwl-docker-extract mpi_simple.cwl crashes at parsing as by default, these extensions are disabled.
$ cwl-docker-extract file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
Traceback (most recent call last):
File "/home/heaucques/Documents/toil/venv3.12/bin/cwl-docker-extract", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 139, in main
run(arg_parser().parse_args(sys.argv[1:]))
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 55, in run
top = cwl.load_document_by_uri(args.input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 183, in load_document_by_uri
return load_document_by_string(doc, real_uri, loadingOptions, id_, load_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 211, in load_document_by_string
return load_document_by_yaml(result, uri, loadingOptions, id_, load_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 227, in load_document_by_yaml
result = cwl_v1_0.load_document_by_yaml(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 21054, in load_document_by_yaml
result, metadata = _document_load(
^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 1038, in _document_load
loader.load(doc, baseuri, loadingOptions, docRoot=baseuri),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 806, in load
raise ValidationException("", None, errors, "*")
schema_salad.exceptions.ValidationException: src/toil/test/cwl/mpi_simple.cwl:3:1: Object `mpi_simple.cwl` is not valid because:
src/toil/test/cwl/mpi_simple.cwl:14:1: the `requirements` field is not valid because:
Expected a <class 'NoneType'> but got list
cwl-docker-extract does not appear to support extensions:
$ cwl-docker-extract --help
usage: cwl-docker-extract [-h] [--dir DIR] [-s] [--container-engine CONTAINER_ENGINE] [--force-download] input
Save container images specified in a CWL document (Workflow or CommandLineTool). For CWL Workflows, all steps will also be searched (recursively).
positional arguments:
input Input CWL document (CWL Workflow or CWL CommandLineTool)
options:
-h, --help show this help message and exit
--dir DIR Directory in which to save images
-s, --singularity Use singularity to pull the image
--container-engine CONTAINER_ENGINE
Specify which command to use to run OCI containers. Defaults to 'docker' (or 'singularity' if --singularity/-s is passed).
--force-download Force pulling a newer container.
Is there a way to enable support for these extensions to allow the parser to pass for these cases?
The text was updated successfully, but these errors were encountered:
stxue1
changed the title
schema_salad.exceptions.ValidationException in parser
Parser does not support cwltool extensions
Jul 16, 2024
stxue1
added a commit
to DataBiosphere/toil
that referenced
this issue
Jul 16, 2024
Given the workflow below:
MPIRequirement
is part of the cwltool extensions. This workflow also works with both cwltool and toil-cwl-runner given the right flags, ex:And the toil equivalent
However, running
cwl-docker-extract mpi_simple.cwl
crashes at parsing as by default, these extensions are disabled.cwl-docker-extract
does not appear to support extensions:Is there a way to enable support for these extensions to allow the parser to pass for these cases?
The text was updated successfully, but these errors were encountered: