Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A Workflow with two steps of the same ExpressionTool fails if the tool inputs an array of files with loadContents #1094

Open
jeremiahsavage opened this issue Mar 29, 2019 · 0 comments
Open

A Workflow with two steps of the same ExpressionTool fails if the tool inputs an array of files with loadContents #1094

jeremiahsavage opened this issue Mar 29, 2019 · 0 comments

Comments

@jeremiahsavage
Copy link
Contributor

jeremiahsavage commented Mar 29, 2019 •
edited
Loading

Expected Behavior

Both wf1.cwl and wf2.cwl pass validation.

Actual Behavior

wf1.cwl validates, while wf2.cwl fails validation.

wf1.cwl contains a single instance of an ExpressionTool, while wf2.cwl contains two instances of that same tool.
The ExpressionTool fails as it loads the contents of an array of Files.

If the input to the ExpressionTool is a File, instead of an array of Files, I get successful validation in a wf2 condition (code not included for this non-array case).

Workflow Code

wf1.cwl:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0

class: Workflow

inputs:
  - id: json_files
    type:
      type: array
      items: File

outputs:
  []

steps:
  - id: run1
    run: expr_tool.cwl
    in:
      - id: json_files
        source: json_files
    out:
      - id: output

wf2.cwl:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0

class: Workflow

inputs:
  - id: json_files
    type:
      type: array
      items: File

outputs:
  []

steps:
  - id: run1
    run: expr_tool.cwl
    in:
      - id: json_files
        source: json_files
    out:
      - id: output

  - id: run2
    run: expr_tool.cwl
    in:
      - id: json_files
        source: json_files
    out:
      - id: output

expr_tool.cwl:

#!/usr/bin/env cwl-runner
cwlVersion: v1.0

requirements:
  - class: InlineJavascriptRequirement

class: ExpressionTool

inputs:
  - id: json_files
    type:
      type: array
      items: File
      inputBinding:
        loadContents: true

outputs:
  - id: output
    type:
      type: array
      items: string

expression: |
  ${
    var output_array = [];
    for (var i = 0; i < inputs.json_files.length; i++) {
      var data = JSON.parse(inputs.json_files[i].contents);
      output_array.push(data['ID']);
    }
    return {'output': output_array};
  }

Successful validation of wf1

(p3) [jeremiah@localhost ~]$ cwltool --validate --debug wf1.cwl 
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
Resolved 'wf1.cwl' to 'file:///home/jeremiah/wf1.cwl'
wf1.cwl is valid CWL.

Failed validation of wf2

(p3) [jeremiah@localhost ~]$ cwltool --validate --debug wf2.cwl
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
Resolved 'wf2.cwl' to 'file:///home/jeremiah/wf2.cwl'
Validation exception
Traceback (most recent call last):
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 630, in __init__
    toolpath_object["run"], loadingContext)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 370, in load_tool
    loadingContext, workflowobj, uri)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 304, in resolve_and_validate_document
    schema.validate_doc(avsc_names, processobj, document_loader, loadingContext.strict)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/schema_salad/schema.py", line 361, in validate_doc
    strip_dup_lineno(bullets(anyerrors, "* ")))
schema_salad.validate.ValidationException: expr_tool.cwl:2:1:  Object `expr_tool.cwl` is not valid because
                      tried `ExpressionTool` but
expr_tool.cwl:9:1:      the `inputs` field is not valid because
expr_tool.cwl:10:5:       item is invalid because
expr_tool.cwl:11:5:         the `type` field is not valid because
                              - tried InputRecordSchema but
expr_tool.cwl:12:7:               * the `type` field is not valid because
                                      the value 'array' is not a valid
                                      enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7:               * invalid field `items`, expected one of: 'fields',
                                  'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:               * invalid field `inputBinding`, expected one of:
                                  'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:           - tried InputEnumSchema but
                                  * missing required field `symbols`
expr_tool.cwl:12:7:               * the `type` field is not valid because
                                      the value 'array' is not a valid
                                      enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7:               * invalid field `items`, expected one of:
                                  'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:               * invalid field `inputBinding`, expected one of:
                                  'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:           - tried InputArraySchema but
expr_tool.cwl:14:7:               invalid field `inputBinding`, expected one of:
                                  'items', 'type', 'label', 'doc', 'name'
Tool definition failed initialization:
Tool definition file:///home/jeremiah/expr_tool.cwl failed validation:
expr_tool.cwl:2:1:    Object `expr_tool.cwl` is not valid because
                        tried `ExpressionTool` but
expr_tool.cwl:9:1:        the `inputs` field is not valid because
expr_tool.cwl:10:5:         item is invalid because
expr_tool.cwl:11:5:           the `type` field is not valid because
                                - tried InputRecordSchema but
expr_tool.cwl:12:7:                 * the `type` field is not valid because
                                        the value 'array' is not a valid
                                        enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7:                 * invalid field `items`, expected one of: 'fields',
                                    'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:                 * invalid field `inputBinding`, expected one of:
                                    'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:             - tried InputEnumSchema but
                                    * missing required field `symbols`
expr_tool.cwl:12:7:                 * the `type` field is not valid because
                                        the value 'array' is not a valid
                                        enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7:                 * invalid field `items`, expected one of:
                                    'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:                 * invalid field `inputBinding`, expected one of:
                                    'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:             - tried InputArraySchema but
expr_tool.cwl:14:7:                 invalid field `inputBinding`, expected one of:
                                    'items', 'type', 'label', 'doc', 'name'
Traceback (most recent call last):
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 630, in __init__
    toolpath_object["run"], loadingContext)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 370, in load_tool
    loadingContext, workflowobj, uri)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 304, in resolve_and_validate_document
    schema.validate_doc(avsc_names, processobj, document_loader, loadingContext.strict)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/schema_salad/schema.py", line 361, in validate_doc
    strip_dup_lineno(bullets(anyerrors, "* ")))
schema_salad.validate.ValidationException: expr_tool.cwl:2:1:  Object `expr_tool.cwl` is not valid because
                      tried `ExpressionTool` but
expr_tool.cwl:9:1:      the `inputs` field is not valid because
expr_tool.cwl:10:5:       item is invalid because
expr_tool.cwl:11:5:         the `type` field is not valid because
                              - tried InputRecordSchema but
expr_tool.cwl:12:7:               * the `type` field is not valid because
                                      the value 'array' is not a valid
                                      enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7:               * invalid field `items`, expected one of: 'fields',
                                  'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:               * invalid field `inputBinding`, expected one of:
                                  'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:           - tried InputEnumSchema but
                                  * missing required field `symbols`
expr_tool.cwl:12:7:               * the `type` field is not valid because
                                      the value 'array' is not a valid
                                      enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7:               * invalid field `items`, expected one of:
                                  'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:               * invalid field `inputBinding`, expected one of:
                                  'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:           - tried InputArraySchema but
expr_tool.cwl:14:7:               invalid field `inputBinding`, expected one of:
                                  'items', 'type', 'label', 'doc', 'name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/main.py", line 675, in main
    tool = make_tool(uri, loadingContext)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/load_tool.py", line 351, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 55, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 535, in __init__
    loadingContext.prov_obj))
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 569, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/home/jeremiah/.virtualenvs/p3/lib/python3.7/site-packages/cwltool/workflow.py", line 636, in __init__
    (toolpath_object["run"], validate.indent(str(vexc))))
cwltool.errors.WorkflowException: Tool definition file:///home/jeremiah/expr_tool.cwl failed validation:
expr_tool.cwl:2:1:    Object `expr_tool.cwl` is not valid because
                        tried `ExpressionTool` but
expr_tool.cwl:9:1:        the `inputs` field is not valid because
expr_tool.cwl:10:5:         item is invalid because
expr_tool.cwl:11:5:           the `type` field is not valid because
                                - tried InputRecordSchema but
expr_tool.cwl:12:7:                 * the `type` field is not valid because
                                        the value 'array' is not a valid
                                        enum_d9cba076fca539106791a4f46d198c7fcfbdb779, expected 'record'
expr_tool.cwl:13:7:                 * invalid field `items`, expected one of: 'fields',
                                    'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:                 * invalid field `inputBinding`, expected one of:
                                    'fields', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:             - tried InputEnumSchema but
                                    * missing required field `symbols`
expr_tool.cwl:12:7:                 * the `type` field is not valid because
                                        the value 'array' is not a valid
                                        enum_d961d79c225752b9fadb617367615ab176b47d77, expected 'enum'
expr_tool.cwl:13:7:                 * invalid field `items`, expected one of:
                                    'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:14:7:                 * invalid field `inputBinding`, expected one of:
                                    'symbols', 'type', 'label', 'doc', 'name'
expr_tool.cwl:11:5:             - tried InputArraySchema but
expr_tool.cwl:14:7:                 invalid field `inputBinding`, expected one of:
                                    'items', 'type', 'label', 'doc', 'name'
(p3) [jeremiah@localhost ~]$

Your Environment

(p3) [jeremiah@localhost ~]$ cwltool --version
/home/jeremiah/.virtualenvs/p3/bin/cwltool 1.0.20190228155703
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jeremiahsavage