Skip to content

Latest commit

 

History

History
934 lines (488 loc) · 62 KB

test-suite-report.md

File metadata and controls

934 lines (488 loc) · 62 KB
Raw

SOLID-TEST-SUITE REPORT

1. SUMMARY by testFile

  • crud-results.json

    CSS

       MAY         {"passed":2,"failed":0,"pending":0,"total":2}
   MUST        {"passed":36,"failed":1,"pending":0,"total":37}
   WPS         {"passed":26,"failed":4,"pending":0,"total":30}
   total       {"passed":64,"failed":5,"pending":0,"total":69}

    ESS

       MAY         {"passed":0,"failed":0,"pending":2,"total":2}
   MUST        {"passed":30,"failed":7,"pending":0,"total":37}
   WPS         {"passed":0,"failed":0,"pending":30,"total":30}
   total       {"passed":30,"failed":7,"pending":32,"total":69}

    NSS

       MAY         {"passed":2,"failed":0,"pending":0,"total":2}
   MUST        {"passed":36,"failed":1,"pending":0,"total":37}
   WPS         {"passed":28,"failed":2,"pending":0,"total":30}
   total       {"passed":66,"failed":3,"pending":0,"total":69}

  • wac-results.json

    CSS

       MUST        {"passed":84,"failed":6,"pending":3,"total":93}
   total       {"passed":84,"failed":6,"pending":3,"total":93}

    ESS

       MUST        {"passed":75,"failed":15,"pending":3,"total":93}
   total       {"passed":75,"failed":15,"pending":3,"total":93}

    NSS

       MUST        {"passed":90,"failed":0,"pending":3,"total":93}
   total       {"passed":90,"failed":0,"pending":3,"total":93}

2. SUMMARY by server

  • CSS

    crud-results.json

       MAY         {"passed":2,"failed":0,"pending":0,"total":2}
   MUST        {"passed":36,"failed":1,"pending":0,"total":37}
   WPS         {"passed":26,"failed":4,"pending":0,"total":30}
   total       {"passed":64,"failed":5,"pending":0,"total":69}

    wac-results.json

       MUST        {"passed":84,"failed":6,"pending":3,"total":93}
   total       {"passed":84,"failed":6,"pending":3,"total":93}

  • ESS

    crud-results.json

       MAY         {"passed":0,"failed":0,"pending":2,"total":2}
   MUST        {"passed":30,"failed":7,"pending":0,"total":37}
   WPS         {"passed":0,"failed":0,"pending":30,"total":30}
   total       {"passed":30,"failed":7,"pending":32,"total":69}

    wac-results.json

       MUST        {"passed":75,"failed":15,"pending":3,"total":93}
   total       {"passed":75,"failed":15,"pending":3,"total":93}

  • NSS

    crud-results.json

       MAY         {"passed":2,"failed":0,"pending":0,"total":2}
   MUST        {"passed":36,"failed":1,"pending":0,"total":37}
   WPS         {"passed":28,"failed":2,"pending":0,"total":30}
   total       {"passed":66,"failed":3,"pending":0,"total":69}

    wac-results.json

       MUST        {"passed":90,"failed":0,"pending":3,"total":93}
   total       {"passed":90,"failed":0,"pending":3,"total":93}

3. UNIT TESTS by testFile and level

  • crud-results.json

    • MAY

      • Alice's pod
      Get RDFa > As JSON-LDCSSESSNSS
      T064 Triplespassedpendingpassed
      Get RDFa > As TurtleCSSESSNSS
      T065 Triplespassedpendingpassed
       
                  MAY results                                 2/0/0/2       0/0/2/2       2/0/0/2

    • MUST

      • Create non-container
      in an existing container > using POSTCSSESSNSS
      T001 creates the resourcepassedpassedpassed
      T002 adds the resource in the container listingpassedpassedpassed
      in an existing container > using PUTCSSESSNSS
      T004 creates the resourcepassedpassedpassed
      T005 adds the resource in the container listingpassedpassedpassed
      in an existing container > using PATCHCSSESSNSS
      T008 creates the resourcepassedfailedpassed
      T009 adds the resource in the container listingpassedfailedpassed
      in a non-existing container > using PUTCSSESSNSS
      T012 creates the resourcepassedpassedpassed
      T013 adds the resource in the container listingpassedpassedpassed
      in a non-existing container > using PATCHCSSESSNSS
      T017 creates the resourcepassedfailedpassed
      T018 adds the resource in the container listingpassedfailedpassed
      • Update
      Using PUT, overwriting plain text with plain textCSSESSNSS
      T022 updates the resourcepassedpassedpassed
      Using PUT, overwriting Turtle with TurtleCSSESSNSS
      T024 updates the resourcepassedpassedpassed
      Using PUT (same Turtle content)CSSESSNSS
      T026 updates the resourcepassedpassedpassed
      Using PATCH to add tripleCSSESSNSS
      T028 updates the resourcepassedfailedpassed
      Using PATCH to replace triple (same content)CSSESSNSS
      T030 updates the resourcepassedpassedpassed
      Using PATCH to replace triple (present)CSSESSNSS
      T032 updates the resourcepassedpassedpassed
      Using PATCH to replace triple (not present)CSSESSNSS
      T034 does not update the resourcepassedpassedpassed
      Using PATCH to remove triple (present)CSSESSNSS
      T036 updates the resourcepassedpassedpassed
      Using PATCH to remove triple (not present)CSSESSNSS
      T038 does not update the resourcepassedpassedpassed
      • Concurrency
      Try to create the same resource, using PUT 10 timesCSSESSNSS
      T040 succeeds exactly oncefailedpassedfailed
      T041 creates the resourcepassedpassedpassed
      T042 body was set by the successful requestpassedpassedpassed
      T043 adds the resource in the container listing exactly oncepassedpassedpassed
      Use PATCH 10 times to add triple to the same resourceCSSESSNSS
      T046 succeeds 10 timespassedfailedpassed
      T047 updates the resourcepassedfailedpassed
      • Alice's storage root
      CSSESSNSS
      T049 is an ldp BasicContainerpassedpassedpassed
      • Delete
      non-containerCSSESSNSS
      T050 deletes the resourcepassedpassedpassed
      T051 removes the resource from the container listingpassedpassedpassed
      non-empty containerCSSESSNSS
      T054 leaves the container with the resource in itpassedpassedpassed
      T055 leaves the resourcepassedpassedpassed
      empty containerCSSESSNSS
      T058 deletes the containerpassedpassedpassed
      • Create container
      in an existing container > using PUTCSSESSNSS
      T060 creates the containerpassedpassedpassed
      T061 adds the resource in the existing container listingpassedpassedpassed
      • Alice's pod
      GET Turtle > As JSON-LDCSSESSNSS
      T066 Triplespassedpassedpassed
      GET Turtle > As TurtleCSSESSNSS
      T067 Triplespassedpassedpassed
      GET JSON-LD > As JSON-LDCSSESSNSS
      T068 Triplespassedpassedpassed
      GET JSON-LD > As TurtleCSSESSNSS
      T069 Triplespassedpassedpassed
       
                  MUST results                                36/1/0/37     30/7/0/37     36/1/0/37

    • WPS

      • Create non-container
      in an existing container > using POSTCSSESSNSS
      T003 emits websockets-pubsub on the containerpassedpendingpassed
      in an existing container > using PUTCSSESSNSS
      T006 emits websockets-pubsub on the containerpassedpendingpassed
      T007 emits websockets-pubsub on the resourcepassedpendingpassed
      in an existing container > using PATCHCSSESSNSS
      T010 emits websockets-pubsub on the containerpassedpendingpassed
      T011 emits websockets-pubsub on the resourcepassedpendingpassed
      in a non-existing container > using PUTCSSESSNSS
      T014 emits websockets-pubsub on the parentpassedpendingpassed
      T015 emits websockets-pubsub on the containerpassedpendingpassed
      T016 emits websockets-pubsub on the resourcepassedpendingpassed
      in a non-existing container > using PATCHCSSESSNSS
      T019 emits websockets-pubsub on the parentpassedpendingpassed
      T020 emits websockets-pubsub on the containerpassedpendingpassed
      T021 emits websockets-pubsub on the resourcepassedpendingpassed
      • Update
      Using PUT, overwriting plain text with plain textCSSESSNSS
      T023 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PUT, overwriting Turtle with TurtleCSSESSNSS
      T025 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PUT (same Turtle content)CSSESSNSS
      T027 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PATCH to add tripleCSSESSNSS
      T029 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PATCH to replace triple (same content)CSSESSNSS
      T031 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PATCH to replace triple (present)CSSESSNSS
      T033 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PATCH to replace triple (not present)CSSESSNSS
      T035 does not emit websockets-pubsub on the resourcefailedpendingpassed
      Using PATCH to remove triple (present)CSSESSNSS
      T037 emits websockets-pubsub on the resourcepassedpendingpassed
      Using PATCH to remove triple (not present)CSSESSNSS
      T039 does not emit websockets-pubsub on the resourcepassedpendingpassed
      • Concurrency
      Try to create the same resource, using PUT 10 timesCSSESSNSS
      T044 emits websockets-pubsub on the container exactly oncefailedpendingfailed
      T045 emits websockets-pubsub on the resource exactly oncefailedpendingfailed
      Use PATCH 10 times to add triple to the same resourceCSSESSNSS
      T048 emits websockets-pubsub on the resource 10 timespassedpendingpassed
      • Delete
      non-containerCSSESSNSS
      T052 emits websockets-pubsub on the containerfailedpendingpassed
      T053 emits websockets-pubsub on the resourcepassedpendingpassed
      non-empty containerCSSESSNSS
      T056 does not emit websockets-pubsub on the containerpassedpendingpassed
      T057 does not emit websockets-pubsub on the resourcepassedpendingpassed
      empty containerCSSESSNSS
      T059 emits websockets-pubsub on the containerpassedpendingpassed
      • Create container
      in an existing container > using PUTCSSESSNSS
      T062 emits websockets-pubsub on the existing containerpassedpendingpassed
      T063 emits websockets-pubsub on the new containerpassedpendingpassed
       
                  WPS results                                 26/4/0/30     0/0/30/30     28/2/0/30

  • wac-results.json

    • MUST

      • From accessTo
      Public accessTo ReadCSSESSNSS
      T070 Shows the correct WAC-Allow header to Bobpassedpassedpassed
      T071 Shows the correct WAC-Allow header to the publicpassedpassedpassed
      Public accessTo Read+Append, Bob accessTo WriteCSSESSNSS
      T072 Shows the correct WAC-Allow header to Bobpassedpassedpassed
      T073 Shows the correct WAC-Allow header to the publicpassedpassedpassed
      • From default
      Public default Read+Append, Bob default WriteCSSESSNSS
      T074 Shows the correct WAC-Allow header to Bobpassedpassedpassed
      T075 Shows the correct WAC-Allow header to the publicpassedpassedpassed
      T076 Shows the Link header containing the aclDocUrl to Alicepassedfailedpassed
      T077 Does not show a Link header containing the aclDocUrl to the publicpendingpendingpending
      • Create
      Using POST to existing containerCSSESSNSS
      T078 Is allowed with accessTo Append accesspassedpassedpassed
      T079 Is allowed with accessTo Write accesspassedpassedpassed
      T080 Is disallowed otherwisepassedfailedpassed
      Using PUT in existing containerCSSESSNSS
      T081 Is allowed with accessTo Write and default Write accesspassedpassedpassed
      T082 Is allowed with accessTo Append and default Write accesspassedpassedpassed
      T083 is disallowed without default Writepassedpassedpassed
      T084 is disallowed without accessTo Write or Appendfailedfailedpassed
      Using PATCH in existing containerCSSESSNSS
      T085 Is allowed with accessTo Write and default Write accesspassedfailedpassed
      T086 Is allowed with accessTo Append and default Write accesspassedfailedpassed
      T087 is disallowed without default Writefailedfailedpassed
      T088 is disallowed without accessTo Write or Appendfailedfailedpassed
      Using PUT in non-existing containerCSSESSNSS
      T089 Is allowed with accessTo Write and default Write accesspassedpassedpassed
      T090 Is allowed with accessTo Append and default Write accesspassedpassedpassed
      T091 is disallowed without default Writepassedpassedpassed
      T092 is disallowed without accessTo Write or Appendfailedfailedpassed
      Using PATCH in non-existing containerCSSESSNSS
      T093 Is allowed with accessTo Write and default Write accesspassedfailedpassed
      T094 Is allowed with accessTo Append and default Write accesspassedfailedpassed
      T095 is disallowed without default Writefailedfailedpassed
      T096 is disallowed without accessTo Write or Appendfailedfailedpassed
      • Update
      Using PUT to appendCSSESSNSS
      T097 Is allowed with accessTo Write access on resourcepassedpassedpassed
      T098 Is disallowed with accessTo Read+Append+Control access on resourcepassedpassedpassed
      T099 Is allowed with default Write access on parentpassedpassedpassed
      T100 Is disallowed with default Read+Append+Control access on parentpassedpassedpassed
      Using PUT to overwriteCSSESSNSS
      T101 Is allowed with accessTo Write access on resourcepassedpassedpassed
      T102 Is disallowed with accessTo Read+Append+Control access on resourcepassedpassedpassed
      T103 Is allowed with default Write access on parentpassedpassedpassed
      T104 Is disallowed with default Read+Append+Control access on parentpassedpassedpassed
      Using PATCH to appendCSSESSNSS
      T105 Is allowed with accessTo Append access on resourcepassedpassedpassed
      T106 Is allowed with accessTo Write access on resourcepassedfailedpassed
      T107 Is disallowed with accessTo Read+Control access on resourcepassedpassedpassed
      T108 Is allowed with default Append access on parentpassedpassedpassed
      T109 Is allowed with default Write access on parentpassedfailedpassed
      T110 Is disallowed with default Read+Control access on parentpassedpassedpassed
      Using PATCH to overwriteCSSESSNSS
      T111 Is allowed with accessTo Read+Write access on resourcepassedpassedpassed
      T112 Is disallowed with accessTo Read+Append+Control access on resourcepassedpassedpassed
      T113 Is disallowed with accessTo Write+Control access on resourcependingpendingpending
      T114 Is allowed with default Read+Write access on parentpassedpassedpassed
      T115 Is disallowed with default Read+Append+Control access on parentpassedpassedpassed
      T116 Is disallowed with default Write+Control access on parentpendingpendingpending
      Update acl fileCSSESSNSS
      T117 Is disallowed with Read+Writepassedpassedpassed
      T118 Is allowed with Controlpassedpassedpassed
      • Read-LoggedIn
      CSSESSNSS
      T119 Is allowed with accessTo Read access on non-container resourcepassedpassedpassed
      T120 Is disallowed with accessTo Append+Write+Control access on non-container resourcepassedpassedpassed
      T121 Is allowed with default Read access on parent of non-containerpassedpassedpassed
      T122 Is disallowed with default Append+Write+Control access on parent of non-containerpassedpassedpassed
      T123 Is allowed with accessTo Read access on container resourcepassedpassedpassed
      T125 Is allowed with default Read access on parent of containerpassedpassedpassed
      T127 Is disallowed to read .acl with Readpassedpassedpassed
      T128 Is allowed to read .acl with Controlpassedpassedpassed
      • ACL doc application
      No access on containerCSSESSNSS
      T129 does not allow GET denied/passedpassedpassed
      T130 does not allow GET denied/noAclDoc/passedpassedpassed
      T131 does not allow GET denied/noAclDoc/noAclDoc.txtpassedpassedpassed
      ACL doc with acl:accessTo on containerCSSESSNSS
      T132 allows GET accessTo/passedpassedpassed
      T133 does not allow GET accessTo/noAclDoc/passedpassedpassed
      T134 does not allow GET accessTo/noAclDoc/noAclDoc.txtpassedpassedpassed
      ACL doc with acl:default on containerCSSESSNSS
      T135 does not allow GET accessTo/passedfailedpassed
      T136 allows GET accessTo/noAclDoc/passedpassedpassed
      T137 allows GET accessTo/noAclDoc/noAclDoc.txtpassedpassedpassed
      • Read
      CSSESSNSS
      T138 Is allowed with accessTo Read access on non-container resourcepassedpassedpassed
      T139 Is disallowed with accessTo Append+Write+Control access on non-container resourcepassedpassedpassed
      T140 Is allowed with default Read access on parent of non-containerpassedpassedpassed
      T141 Is disallowed with default Append+Write+Control access on parent of non-containerpassedpassedpassed
      T142 Is allowed with accessTo Read access on container resourcepassedpassedpassed
      T144 Is allowed with default Read access on parent of containerpassedpassedpassed
      T146 Is disallowed to read .acl with Readpassedpassedpassed
      T147 Is allowed to read .acl with Controlpassedpassedpassed
      • Read-Public
      CSSESSNSS
      T148 Is allowed with accessTo Read access on non-container resourcepassedpassedpassed
      T149 Is disallowed with accessTo Append+Write+Control access on non-container resourcepassedpassedpassed
      T150 Is allowed with default Read access on parent of non-containerpassedpassedpassed
      T151 Is disallowed with default Append+Write+Control access on parent of non-containerpassedpassedpassed
      T152 Is allowed with accessTo Read access on container resourcepassedpassedpassed
      T154 Is allowed with default Read access on parent of containerpassedpassedpassed
      T156 Is disallowed to read .acl with Readpassedpassedpassed
      T157 Is allowed to read .acl with Controlpassedpassedpassed
      • Delete
      CSSESSNSS
      T158 Is allowed with accessTo Write access on resourcepassedpassedpassed
      T159 Is disallowed with accessTo Read+Append+Control access on resourcepassedpassedpassed
      T160 Is allowed with default Write access on parentpassedpassedpassed
      T161 Is disallowed with default Read+Append+Control access on parentpassedpassedpassed
      • Alice's storage root
      CSSESSNSS
      T162 has an ACLpassedpassedpassed
       
                  MUST results                                84/6/3/93     75/15/3/93    90/0/3/93

4. ERROR REPORT

T008 MUST > crud-results.json > ["Create non-container","in an existing container","using PATCH"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T009 MUST > crud-results.json > ["Create non-container","in an existing container","using PATCH"]

T017 MUST > crud-results.json > ["Create non-container","in a non-existing container","using PATCH"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T018 MUST > crud-results.json > ["Create non-container","in a non-existing container","using PATCH"]

T028 MUST > crud-results.json > ["Update","Using PATCH to add triple"]

T035 WPS > crud-results.json > ["Update","Using PATCH to replace triple (not present)"]

  • CSS

    Error: expect(received).not.toEqual(expected) // deep equality

    Expected: not ArrayContaining ["pub http://localhost:3000/solid-crud-tests-1615739594573/exists/exists5.ttl"]

    Received: ["protocol solid-0.1", "warning Unrecognized message format: auth ", "warning Unrecognized message format: dpop ", "ack http://localhost:3000/solid-crud-tests-1615739594573/exists/exists5.ttl", "pub http://localhost:3000/solid-crud-tests-1615739594573/exists/exists5.ttl"]

T040 MUST > crud-results.json > ["Concurrency","Try to create the same resource, using PUT 10 times"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

  • NSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

T044 WPS > crud-results.json > ["Concurrency","Try to create the same resource, using PUT 10 times"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

  • NSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

T045 WPS > crud-results.json > ["Concurrency","Try to create the same resource, using PUT 10 times"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

  • NSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 1

    Received: 10

T046 MUST > crud-results.json > ["Concurrency","Use PATCH 10 times to add triple to the same resource"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 10

    Received: 0

T047 MUST > crud-results.json > ["Concurrency","Use PATCH 10 times to add triple to the same resource"]

T052 WPS > crud-results.json > ["Delete","non-container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: ArrayContaining ["ack http://localhost:3000/solid-crud-tests-1615739615460/exists/", "pub http://localhost:3000/solid-crud-tests-1615739615460/exists/"]

    Received: ["protocol solid-0.1", "warning Unrecognized message format: auth ", "warning Unrecognized message format: dpop ", "ack http://localhost:3000/solid-crud-tests-1615739615460/exists/"]

T076 MUST > wac-results.json > ["From default","Public default Read+Append, Bob default Write"]

T080 MUST > wac-results.json > ["Create","Using POST to existing container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T084 MUST > wac-results.json > ["Create","Using PUT in existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T085 MUST > wac-results.json > ["Create","Using PATCH in existing container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T086 MUST > wac-results.json > ["Create","Using PATCH in existing container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T087 MUST > wac-results.json > ["Create","Using PATCH in existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T088 MUST > wac-results.json > ["Create","Using PATCH in existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T092 MUST > wac-results.json > ["Create","Using PUT in non-existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T093 MUST > wac-results.json > ["Create","Using PATCH in non-existing container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T094 MUST > wac-results.json > ["Create","Using PATCH in non-existing container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T095 MUST > wac-results.json > ["Create","Using PATCH in non-existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T096 MUST > wac-results.json > ["Create","Using PATCH in non-existing container"]

  • CSS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 205

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 201

T106 MUST > wac-results.json > ["Update","Using PATCH to append"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T109 MUST > wac-results.json > ["Update","Using PATCH to append"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: "2xx"

    Received: "4xx"

T135 MUST > wac-results.json > ["ACL doc application","ACL doc with acl:default on container"]

  • ESS

    Error: expect(received).toEqual(expected) // deep equality

    Expected: 403

    Received: 200