Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parser does not support cwltool extensions #309

Open
stxue1 opened this issue Jul 16, 2024 · 0 comments
Open

Parser does not support cwltool extensions #309

stxue1 opened this issue Jul 16, 2024 · 0 comments

Comments

@stxue1
Copy link

stxue1 commented Jul 16, 2024

Given the workflow below:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
$namespaces:
  cwltool: "http://commonwl.org/cwltool#"

doc: |
  Trivial MPI test that prints the process IDs of each of the parallel
  processes. Requires Python (but you have cwltool running, right?)
  and an MPI implementation.

baseCommand: python
requirements:
  cwltool:MPIRequirement:
    processes: 2
arguments: [-c, 'import os; print(os.getpid())']
inputs: []
outputs:
  pids:
    type: stdout

MPIRequirement is part of the cwltool extensions. This workflow also works with both cwltool and toil-cwl-runner given the right flags, ex:

$ export PATH=$PATH:/home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/ # where fake_mpi_run.py exists
$ cwltool --enable-dev --enable-ext --mpi-config-file /home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/fake_mpi.yml file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
INFO /home/heaucques/Documents/toil/venv3.12/bin/cwltool 3.1.20240508115724
INFO [job mpi_simple.cwl] /tmp/1ez0it5c$ fake_mpi_run.py \
    --num \
    2 \
    --no-fail \
    python \
    -c \
    'import os; print(os.getpid())' > /tmp/1ez0it5c/c83195d490fa84f03f0cb9b142c6f385d736bd94
INFO [job mpi_simple.cwl] completed success
{
    "pids": {
        "location": "file:///home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94",
        "basename": "c83195d490fa84f03f0cb9b142c6f385d736bd94",
        "class": "File",
        "checksum": "sha1$010210c4a6298629ae0653165e70c393bc831cad",
        "size": 12,
        "path": "/home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94"
    }
}INFO Final process status is success

And the toil equivalent

However, running cwl-docker-extract mpi_simple.cwl crashes at parsing as by default, these extensions are disabled.

$ cwl-docker-extract file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/bin/cwl-docker-extract", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 139, in main
    run(arg_parser().parse_args(sys.argv[1:]))
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 55, in run
    top = cwl.load_document_by_uri(args.input)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 183, in load_document_by_uri
    return load_document_by_string(doc, real_uri, loadingOptions, id_, load_all)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 211, in load_document_by_string
    return load_document_by_yaml(result, uri, loadingOptions, id_, load_all)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 227, in load_document_by_yaml
    result = cwl_v1_0.load_document_by_yaml(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 21054, in load_document_by_yaml
    result, metadata = _document_load(
                       ^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 1038, in _document_load
    loader.load(doc, baseuri, loadingOptions, docRoot=baseuri),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 806, in load
    raise ValidationException("", None, errors, "*")
schema_salad.exceptions.ValidationException: src/toil/test/cwl/mpi_simple.cwl:3:1:   Object `mpi_simple.cwl` is not valid because:
src/toil/test/cwl/mpi_simple.cwl:14:1:   the `requirements` field is not valid because:
                                           Expected a <class 'NoneType'> but got list

cwl-docker-extract does not appear to support extensions:

$ cwl-docker-extract --help
usage: cwl-docker-extract [-h] [--dir DIR] [-s] [--container-engine CONTAINER_ENGINE] [--force-download] input

Save container images specified in a CWL document (Workflow or CommandLineTool). For CWL Workflows, all steps will also be searched (recursively).

positional arguments:
  input                 Input CWL document (CWL Workflow or CWL CommandLineTool)

options:
  -h, --help            show this help message and exit
  --dir DIR             Directory in which to save images
  -s, --singularity     Use singularity to pull the image
  --container-engine CONTAINER_ENGINE
                        Specify which command to use to run OCI containers. Defaults to 'docker' (or 'singularity' if --singularity/-s is passed).
  --force-download      Force pulling a newer container.

Is there a way to enable support for these extensions to allow the parser to pass for these cases?

@stxue1 stxue1 changed the title schema_salad.exceptions.ValidationException in parser Parser does not support cwltool extensions Jul 16, 2024
DailyDreaming pushed a commit to DataBiosphere/toil that referenced this issue Jul 19, 2024
* Prepull containers (docker and singularity) by default

* Don't prepull containers if extensions are enabled
See common-workflow-language/cwl-utils#309

* Remove unused import

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant