Support W3C EARL schema for test results

[rjmartell]

The W3C Evaluation and Report Language (EARL) 1.0 Schema defines a standard RDF vocabulary for expressing test results. Adopting this for all test suites enables a more consistent approach for test reporting. A developer guide is available.

Note: The Evaluation and Repair Tools Working Group (ERT WG) has not been rechartered, so the status of EARL is unclear.

[rjmartell]

This could be implemented as a custom test listener for use in TestNG-based suites. CTL results could be transformed via XSLT stylesheets (to RDF/XML format).

[bermud]

@keshav-nangare,please take a look at this commit and see if it fits the general conformance testing model. Also please take a look at the wiki notes.

[keshavnangare]

@rjmartell
Just to know, have you added the EarlReportListener into the TestNGExecutor as listener? because it did not generate the output file.

Or is there any other way to invoke the EarlReportListener. I have executed but I didn't get the output file.

Can you please update me how to invoke this EarlReportListener?

[bermud]

@richard and @keshav-nangare, I have some comments, based on the wiki and the attached file.

• I cannot find where “earl:outcome” is declared
• An Assertion is a statement that has a subject, predicate and object. We are missing the predicated and object
• the predicate provides the result of the assertion. There are two options fails and pass. For example: rdf:predicate earl:passes;. Look at the W3C Example1 or Example 2

a :Assertion;
             rdf:subject <wms-server ..>;
             rdf:predicate  [
                 :validity :Pass;
                 a :ResultProperty;              
                 :date "2001-03-14" 
                 ... ];

• The object is a test case. Instead of doing;

 <earl:Assertion rdf:about="assert-29">
    <earl:test>
      <earl:TestCase rdf:about="org/opengis/cite/iso19136/data/spatial/CurveTests#curveHasValidCRS">
        <dct:description>See ISO 19136: 9.10, 10.1.3.2; ISO 19107: 6.2.2.17</dct:description>
        <dct:title>curveHasValidCRS</dct:title>
      </earl:TestCase>
    </earl:test>

we can do

 <earl:Assertion rdf:about="assert-29">
 ...
    </rdf:object>
      <earl:TestCase rdf:about="org/opengis/cite/iso19136/data/spatial/CurveTests#curveHasValidCRS">
        <dct:description>See ISO 19136: 9.10, 10.1.3.2; ISO 19107: 6.2.2.17</dct:description>
        <dct:title>curveHasValidCRS</dct:title>
      </earl:TestCase>
    </rdf:object>
 ...

A subject is declared but details are never given (e.g <earl:subject pdf:resource=“f2603785-2c6b-4acf-85e9-ea82e6eda3ab”/>. I guess this should be the application being tested or the file. It should have its own properties (e.g. name, url).

earl.rdf.zip

[rjmartell]

Those example are very old--they're for EARL 0.95 (from 2001). We're using the latest (last call) WD for EARL 1.0. The relevant triple is:

[earl:TestResult] earl:outcome [earl:OutcomeValue]

There are five instances of OutcomeValue declared in the model (earl:passed, earl:failed, etc.). There are several examples in the Developer Guide for EARL 1.0.

[rjmartell]

There are a few other related specs that can be used to supply additional information about a resource such as earl:TestResult or earl:TestSubject:

• HTTP Vocabulary in RDF 1.0 (W3C Working Draft)
• Pointer Methods in RDF 1.0 (W3C Working Draft)
• Representing Content in RDF 1.0 (W3C Working Draft)

For example: including request and response messages for failed tests; or adding pointers to specific fragments in a resource representation.

[bermud]

Thank you for providing the correct version.

About the result,
I agree should be: <earl:outcome rdf:resource="http://www.w3.org/ns/earl#passed"/>
I think all the declared outcomes fit our model: earl:Pass, earl:Fail, earl:CannotTell, earl:NotApplicable, earl:NotTested an earl:NotTested

About the types of Subject
It says it is recommended to not use earl:TestSubject, but rather earl:Software, cnt:Content,http:Response,or foaf:Document
So for:

• testing web service we should use earl:Software
• for concrete http call we should use s http:Response. Example 2.4 provide details that we can use to capture the response information
• for encodings we should use cnt:Content

Providing more details about a subject, if needed, then it seems we can use doap elements, like Example 13 .

[bermud]

Please follow issue #81 for more work related to this item.