Test Metadata Draft Spec

Test Metadata Draft Specification

Please note this is a draft, subject to change. Please leave any comments here: https://github.com/crs4/life_monitor/issues/15.

The Life Monitor (LM) service is being developed to serve as a testing platform for workflows registered in the WorkflowHub (WFHub). The Workflow RO-Crate spec allows for an optional test subdirectory to hold tests. This document describes how to structure the metadata within the test directory so that it can be consumed by the LM.

File and directory layout

If present in the RO-Crate, the test directory must contain a file named as follows:

test-metadata.json

In the simplest cases, only this file will be present; in the general case, the test directory might also include other files or directories, which would be referenced (directly or indirectly) by test-metadata.json.

Use cases

Several scenarios are possible, according to whether:

• the workflow has been registered to WFHub as
  • a Workflow RO-Crate (including the test directory)
  • an individual file
• the workflow has been registered via
  • file upload
  • remote URL
• the workflow is tested
  • externally: the workflow is already being tested on an external service (e.g., a Jenkins instance), and LM acts as an intermediary to collect and report test results
  • internally: a test project defined according to the test metadata is run on a CI (e.g., Jenkins) instance managed by LM

The main use case we are considering is the one where the workflow has been registered to WFHub as an RO-Crate and is tested internally.

From the LM perspective it does not matter whether the workflow has been registered via an upload or a link to an external resource, since it's always possible to get it as an RO-Crate from the WFHub. In the case of a workflow registered as an RO-Crate, the test metadata can be provided by the crate creator in its final form as described here. For workflows registered as individual files, the user would need a way to specify the extra test data and metadata, which would then be included in the RO-Crate by WFHub. WFHub could enable this in several ways:

• when only test-metadata.json is needed (e.g., in the case of external testing), an extra file upload (or remote URL) slot could be provided.
• when a full test directory tree is needed:
  • file upload is still viable, e.g., as a tar or zip archive
  • the remote URL slot could point to the test directory on the repository

As far as this spec is concerned, the only relevant distinction is between internal and external testing, since the assumption is that a Workflow RO-Crate containing test information can always be obtained from the WFHub. However, the case where the workflow has been registered as an RO-Crate is already supported by WFHub, since in this case the RO-Crate is built by the user that registers the workflow and can be structured as specified here.

Test specification examples

External test service

{
    "tmpformat": "ro/workflow/test-metadata/0.1",
    "@id": "test-metadata.json",
    "test": [
	{
  	    "name": "dtests",
  	    "instance": [
    		{
      		    "name": "dtests",
      		    "service": {
        		"type": "jenkins",
        		"url": "http://172.30.10.90:8080/",
			"resource": "job/dtests/"
      		    }
    		}
  	    ],
  	    "definition": {}
	}
    ]
}

Test definition (internal testing)

Embedded definition (payload):

{
    "tmpformat": "ro/workflow/test-metadata/0.1",
    "@id": "test-metadata.json",
    "test": [
	{
  	    "name": "dtests",
  	    "instance": [
  	    ],
  	    "definition": {
		"test_engine": {
		    "type": "planemo",
		    "version": ">=0.70"
		},
		"payload": {
		    "content": "- doc: my test\n  job: my_job.yml\n  outputs:\n	out1:\n  	path: test-data/exp1.txt\n	out2:\n  	path: test-data/exp2.txt\n",
		    "encoding": null
		}
            }
	}
    ]
}

Definition in a separate file (path):

{
    "tmpformat": "ro/workflow/test-metadata/0.1",
    "@id": "test-metadata.json",
    "test": [
	{
  	    "name": "dtests",
  	    "instance": [
  	    ],
  	    "definition": {
		"test_engine": {
		    "type": "planemo",
		    "version": ">=0.70"
		},
		"path": "path relative to the directory containing this file"
            }
	}
    ]
}

Both test service and test definition are present

Shown with separate file definition, but embedded payload is also possible.

{
    "tmpformat": "ro/workflow/test-metadata/0.1",
    "@id": "test-metadata.json",
    "test": [
	{
  	    "name": "dtests",
  	    "instance": [
    		{
      		    "name": "dtests",
      		    "service": {
        		"type": "jenkins",
        		"url": "http://172.30.10.90:8080/",
			"resource": "job/dtests/"
      		    }
    		}
  	    ],
  	    "definition": {
		"test_engine": {
		    "type": "planemo",
		    "version": ">=0.70"
		},
		"path": "path relative to the directory containing this file"
            }
	}
    ]
}