Implement a subset of the Common Workflow Language

[nsoranzo]

Adding support for a large subset of CWL v1.0.2, v1.1 and v1.2 . This is a group work, led by @jmchilton with contributions from @mr-c , @mvdbeek , @hmenager and myself.
This effort wouldn't have been possible without the support of ELIXIR which sponsored this project at BioHackathon Europe in 2018, 2019, 2020 and 2021.

CWL Support (Tools):

• Implemented integer, long, float, double, boolean, string, File, Directory, "null", Any, as well as records and arrays thereof. There are two approaches to handling more complex parameters discussed here (Create a Design Document Discussing CWL to Galaxy State Mapping common-workflow-lab/galaxy#59).
• secondaryFiles that are actual Files are implemented, secondaryFiles containing directories are not yet implemented.
• InlineJavascriptRequirement are support to define output files (see test_cat3 test case).
• EnvVarRequirements are supported (see the test_env_tool1 and test_env_tool2 test cases).
• Expression tools are supported (see parseInt-tool test case).
• Shell tools are also support (see record output test case).
• Default File values are very un-Galaxy and have been hacked into work with Tools - they still don't work with workflows.
• Partial Docker support - this supports the most simple and common pullFrom semantics but not additional ways to fetch containers or additional options such as output directory configuration (https://github.com/common-workflow-lab/galaxy/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20Docker). Additionally, Galaxy mounts the inputs and outputs where it wants instead of CWL required mount points - this needs to be fixed for the conformance tests but may not matter much in practice (I'm not sure).

CWL Support (Workflows):

• Simple connections and tool execution.
• Overriding tool input defaults via literal values and simple expressions.
• MultipleInputFeatureRequirements to glue together multiple file inputs into a File[] or multiple File[] into a single flat File[]. (nested merge is still a TODO).
• Simple scatter semantics for Files and non-Files (e.g. count-lines3).
• Simple subworkflows (e.g. count-lines10).
• Simple valueFrom expressions (e.g. step-valueFrom and step-valueFrom2). This work doesn't yet model non-tool parameters to steps - for complex valueFrom expressions like in step-valueFrom3 do not work yet.

Remaining Work

The work remaining is vast and will be tracked at https://github.com/common-workflow-lab/galaxy/issues for the time being.

Implementation Notes:

Tools:

• Non-File CWL outputs are represented as expression.json files. Traditionally Galaxy hasn't supported non-File outputs from tools but CWL Galaxy has work in progress on bringing native Galaxy support for such outputs [STABLE] Fix purge quota adjustment via manager functionality. #27.
• CWL secondary files are just normal datasets with extra files stored in __secondary_files__ directory in the dataset's extra_files_path directory and indexed in a file called __secondary_files_index.json in extra_files_path. The upload tools has been augmented to allow attaching arbitrary extra files as a tar file to support getting data into this format initially. CWL requires staging files to include their parent File's basename - but tools describe inputs as just the extension. I'm not sure which way Galaxy should store secondary_files in its objectstore - just with the extension or with the basename and extension - both options are implemented and can be swapped by setting the boolean STORE_SECONDARY_FILES_WITH_BASENAME in galaxy.tools.cwl.util.
• CWL Directory types are datasets of a new type "directory" implemented earlier in this branch.
• The tool execution API has been extended to add a inputs_representation parameter that can be set to "cwl" now. The cwl representation for running tools corresonding to the CWL job json format with {class: "File: path: "/path/to/file"} inputs replaced with {"src": "hda", "id": "<dataset_id>"}. Code for building these requests for CWL job json is available in the test class.
• Since the CWL <-> Galaxy parameter translation may change over time, for instance if Galaxy develops or refines parameter classes - CWL state and CWL state version is tracked in the database and hopefully for reruns, etc... we could update the Galaxy state from an older version to a new one.
• CWL allows output parameters to be either File or non-File and determined at runtime, so galaxy.json is used to dynamically adjust output extension as needed for non-File parameters.

Workflows:

• This work serializes embedded and referenced tools into the database - this will allow reuse and tracing without require the path to exist forever on the filesystem - this will have problems with default file references in workflows.
• Implements re-mapping CWL workflow connections to Galaxy input connections.
• Fix tool serialization for jobs for path-less tools (such as embedded tools).
• Hack tool state during workflow import for CWL.
• The sort of dynamic shaping of inputs CWL allows has required enhancing Galaxy's map/reduce stuff to allow mapping over dynamic collections that don't yet exist at the time of tool execution and need to be created on the fly. This commit creates them as HDCAs - but likely they should be something else that doesn't appear in the history panel.
• Multi-input scattering but only scatterMethod == "dotproduct" is currently support. Other scatter methods (nested_crossproduct and flatcross_product) are not used by workflows in GA4GH challenge.

Implementation Description:

The reference implementation Python library (mainly developed by Peter Amstutz) is used to load tool files ending with .json or .cwl and proxy objects are created to adapt these tools to Galaxy representations. In particular input and output descriptions are loaded from the tool.

When the tool is submitted, a special specialized tool class is used to build a cwltool compatible job description from the supplied Galaxy inputs and the CWL reference implementation is used to generate a CWL reference implementation Job object. A command-line is generated from this Job object.

As a result of this - Galaxy largely does not need to worry about the details of command-line adapters, expressions, etc....

Galaxy writes a description of the CWL job that it can reload to the job working directory. After the process is complete (on the Galaxy compute server, but outside the Docker container) this representation is reloaded and the dynamic outputs are discovered and moved to fixed locations as expected by Galaxy. CWL allows for much more expressive output locations than Galaxy, for better or worse, and this step uses cwltool to adapt CWL to Galaxy outputs.

Currently all File outputs are sniffed to determined a Galaxy datatype, CWL allows refinement on this and this remains work to be done.

1. CWL should support EDAM declaration of types and Galaxy should provide a mapping to core datasets to skip sniffing is types are found.
2. For finer grain control within Galaxy, extensions to CWL should allow setting actual Galaxy output types on outputs. (Distinction between fastq and fastqsanger in Galaxy is very important for instance.)

Implementation Links:

Hundreds of commits have been rebased into this one and so the details of individual parts of the implementation and how they built on each other are not entirely clear. To see the original ideas behind individual features - here are some relevant links:

• Implement merge_nested link semantics for workflow steps (a903abd).
• Implement subworkflows in CWL (9933c3c)
• MultipleInputFeatureRequirements:
  • Second attempt: ed8307f
  • First attempt: ae11f56
• Basic, implicit dotproduct scattering of workflows - d1ad64e.
• Simple input StepInputExpressionRequirements - 819a27b
• StepInputExpressionRequirements for multiple inputs - 5e7f622
• Record Types in CWL - e6be28a
• Rework original approach at mapping CWL state to tool state - 669ea55
• Rework approach at mapping CWL state to tool state again to use "FieldTypeToolParameter"s - implements default values, optional parameters, and union types for workflow inputs. d1ca22f
• Initial tracking of "cwl_filename" for CWL jobs (67ffc55).
• Reworked secondary file staging, implement testing and indexing of secondary files - 03d1636.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to galaxy-committers@lists.galaxyproject.org.

[jmchilton]

I'd feel more comfortable about this if we were choosing between the two based on the job type in some way.