More granular supported_filesystem_protocols in service info

[jaeddy]

Currently this is just an array:

      supported_filesystem_protocols:
        type: array
        items:
          type: string
        description: |-
          The filesystem protocols supported by this service, currently these may include common
          protocols such as 'http', 'https', 'sftp', 's3', 'gs', 'file', 'synapse', or others as
          supported by this service.

However, the WES server might (is likely to?) support different protocols than the environment where individual tasks are executed. For example, the Broad Cromwell WES we used for the Toronto Testbed Demo was set up on an EC2 instance and could receive workflow contents (descriptors, parameters) via http; however, tasks were run via PAPI and thus required inputs/outputs as GS bucket URLs. For v1.0 of the spec, we're also proposing to store workflow and task logs at accessible (to the client) URLs, and so some protocol hint might be useful for those as well.

I'm thinking we could change this property to an object with keys indicating supported protocols for workflows, tasks, and logs, respectively:

{"workflows": ["http", "file"], "tasks": ["gs"], "logs": ["gs"]}

This might still be an oversimplification (e.g., if a WES endpoint supports multiple task execution backends, then each might support a different fs protocol...) — but it would at least provide an initial barrier to prevent a client from submitting incompatible jobs.

[briandoconnor]

All http for Basel

[geoffjentry]

@briandoconnor should we remove the Due July 2018 label?

[jaeddy]

Revisiting this....

I propose something along the lines of the following: add the supported_schemes property to the ServiceInfo response body (and mark supported_filesystem_protocols for deprecation). Rather than specifying at which level a scheme/protocol is used by the WES implementation, this approach focuses on what will be accessed/retrieved.

The WES service doesn't need to describe the full mechanics as to how and where it uses the supported schemes, but it should offer a guarantee that run requests that include allowable combinations of those schemes will succeed (or at least not cause an error).

  ServiceInfo:
    type: object
    properties:
      supported_schemes:
        $ref: '#/definitions/SupportedSchemes'

...

  SupportedSchemes:
    type: object
    properties:
      workflow_url:
        description: >-
          The URL schemes (protocols) supported by this service to access or
          retrieve the main descriptor/document for a workflow. These are most
          likely to be 'https' (GitHub) or 'file' (in cases when the 
          'workflow_attachment' is used to upload descriptors), but others
          are possible.
        type: array
        items:
          type: string
      workflow_param:
        description: >-
          The URL schemes (protocols) supported by this service to access or
          retrieve objects specified in the workflow parameters. Currently 
          these may include common protocols using the terms 'http', 'https',
          'sftp', 's3', 'gs', 'file', or 'synapse', but others are possible 
          and the terms beyond these core protocols are currently not fixed.

          Note that the 'drs' scheme indicates that this service can use the
          Data Repository Service (DRS) API to resolve a DRS URI to one of
          several different URL types, includinding (but not limited to) the
          examples above.
        type: array
        items:
          type: string
      workflow_logs:
        description: >-
          The URL schemes (protocols) used by this service to store and
          provide access to workflow log files (e.g., stdout, stderr). These 
          may include common protocols using the terms 'http', 'https',
          'sftp', 's3', 'gs', but others are possible.

Will work on a PR soon, but happy to get other thoughts on this.

[susheel]

+1 @jaeddy A WES service may report that they are not able to access workflow_* due to an unsupported access protocol, as such I think we need a special Error status for this condition.