Broadcasting an API implementation's optional capabilities

[uniqueg]

Problem

There are situations where managers of GA4GH API products deem certain functionalities "nice to have", but feel that they may not be necessary for all implementations or that they would unduly raise the bar for new implementers. In such cases, product managers/contributors may recommend or even specify support for such features if implementers (or administrators of individual instances) choose to adopt/provide them.

Currently, there is no common method for service instances to broadcast any such optional capabilities to make clients aware of such support.

An example

A Driver Project may be interested in the GA4GH Task Execution Service (TES) API providing dedicated support for Crypt4GH-encrypted inputs. Such support may require changes to the TES specification, e.g., additional properties in the task resource creation schema). TES product managers may agree that specific Crypt4GH support is useful but find it unreasonable to mandate that all TES implementations provide such support. They would now have at least two options to provide/specify support:

• Add the necessary fields to the existing task resource creation schema nad make them optional (or required only upon some condition)
• Publish a Crypt4GH extension of the TES specification

Either way, if TES implementers choose to provide Crypt4GH support, there is currently no common way of letting clients know about this support.

A possible solution

Broadcast optional, but clearly defined capabilities via an API service instance's GET /service-info endpoint. To make it easy for clients to consume (identify and understand) such capabitlies, at least the following requirements have to be met:

1. A controlled vocabulary of capabilities. These may be product-specific or cross-product; a given capability may also appear to be product-specific at first, but turn out later to be applicable to other products as well. They may also be hierarchical. An example could be crypt4gh.
2. A schema for specifying capabilities in the service info API. This could look, e.g., something like this:

components:
  ...
  schemas:
    Service:
      description: 'GA4GH service'
      type: object
      ...
      properties:
        ...
        capabilities:
          $ref: '#components/schemas/Capabilities'
    ...
    Capabilities:
      type: object
      description: Information on the support of optional capabilities of this service.
      additionalProperties:
        $ref: '#/components/schemas/Capability'
    Capability:
      type: object
      required:
        - supported
      properties:
        supported:
          type: boolean
          description: Whether the capability is supported by the service instance.
      anyOf:
        - properties:
            documentationUrl:
              type: string
              description: URL to documentation of this capability.
        - properties:
            specificationUrl: 
              type: string
              description: URL to an API specification supporting this capability.

This defines an interface for any named Capability, listed under property capabilities. The generic interface would require, for any listed capability, at least information on whether that capability is supported and either or both of a documentationUrl and/or a specificationUrl (in case there is a specific extension defined).

Then, in TES, one could further specify the general interface (this shows the general pattern of how the service info's Capability schema could be extended, if necessary; details would of course differ for each capability):

components:
  ...
  schemas:
    CapabilityCrypt4GH:
      allOf:
        - '$ref': https://raw.githubusercontent.com/ga4gh-discovery/ga4gh-service-info/v1.0.0/service-info.yaml#/components/schemas/Capability
        - properties:
            some_tes_crypt4gh_specific_property:
              type: string
              description: Some capability-specific parameter

Note that the suggested (ad hoc) solution does not support nesting of capabilities. For example, crypt4gh-decrypt and crypt4gh-encrypt capabilities would have to be listed separately, rathan than as:

crypt4gh:
  decrypt:
    ...
  encrypt:
    ...

If support for nesting is desired, this will need additional work and decisions on whether there is a pre-defined number of nestings, whether nesting is always required and so on.

Possible alternatives

I was considering raising this issue in the Service Info API. However, I am not sure whether the service info is definitely the way to go here. Also, I feel that a solution should be dicussed and agreed upon by a wide range of GA4GH API product managers.

Additional context

Broadcasting capabilities via the /service-info endpoint would have the additional strong benefit that capabilities would become discoverable through Service Registry API implementations.

[uniqueg]

As discussed in the meeting, here are more possible use cases. These are all from the Cloud WS - perhaps others can add more for other products:

• Support operations for posting and/or putting and/or patching and/or deleting resources (DRS,TRS; for deletions also WES, TES)
• Accept DRS URIs for inputs (TES, WES)
• Accept TRS URIs for containers/workflows (TES, WES)
• Advertise supported file transfer protocols (s3, gs, az, ftp, etc.) (WES, TES)
• Process Crypt4GH-encrypted objects (TES, DRS)
• Return specific log format (TES, WES)
• Validating a workflow and inputs ("dry run") (WES)
• Indicate that a WES uses or can use a TES backend (WES)
• Support for Confidential Computing (TES)
• Support task/workflow run status callbacks (TES, WES)
• Support re-running/-starting task/workflow runs (TES, WES)
• Support for sending passports in request bodies (DRS; should never have made it to the core spec, IMO)
• Make support for workflow attachments optional (WES)
• Capture usage statistics on resources (DRS, TRS)
• Advertise supported container flavors (OCI, Singularity, etc.) (TES)
• Support for service-to-service communication (e.g., for federated learning use cases) (TES)
• Support for persistent data volumes (TES)

Not saying that all (or any) of them should necessarily be added as extensions. That would need to be discussed on an individual basis.

@ianfore @mbaudis @andrewyatz