Conformance Test Mapping to WPT

[lgrahl]

This is a mapping of all conformance tests to the W3C web platform tests of this PR:

• init.PC: RTCPeerConnection-constructor, RTCPeerConnection.length
• init.PCWebkit: discarded, reason: not a conformance check
• label.label001: RTCPeerConnection-createDataChannel, createDataChannel with no argument should throw TypeError
• label.label002: RTCPeerConnection-createDataChannel, Zero length label and protocol option should be transmitted via DCEP
• label.label003: RTCPeerConnection-createDataChannel, createDataChannel with label lone surrogate should succeed
• label.label004: discarded, reason: not an edge case
• label.label005: RTCDataChannel-dcep, Maximum length label and protocol option should be transmitted via DCEP
• label.label006: RTCDataChannel-dcep, createDataChannel with too long label should throw TypeError (also tests for negotiated channels)
• label.label007: RTCDataChannel-dcep, Zero length label and protocol option should be transmitted via DCEP
• label.label008: RTCPeerConnection-createDataChannel, createDataChannel with same label used twice should not throw
• label.label009: discarded, reason: not much of a difference to label.008
• label.label010: discarded, reason: invalid due to spec changing to USVString. Valid test cases for null and undefined are in RTCPeerConnection-createDataChannel, createDataChannel with label ... should succeed.
• label.label011: RTCDataChannel-dcep, Maximum length label and protocol option (3 byte unicode) should be transmitted via DCEP
• dict.dict001: RTCDataChannel-dcep, ... and unreliable (maxPacketLifeTime=null) channel should be created via DCEP
• dict.dict002: RTCDataChannel-dcep, ... and unreliable (maxPacketLifeTime=65535) channel should be created via DCEP
• dict.dict003: RTCDataChannel-dcep, ... and unreliable (maxPacketLifeTime=4294967295) channel should be created via DCEP
• dict.dict004: RTCDataChannel-dcep, ... and unreliable (maxRetransmits=null) channel should be created via DCEP
• dict.dict005: RTCDataChannel-dcep, ... and unreliable (maxRetransmits=65535) channel should be created via DCEP
• dict.dict006: RTCDataChannel-dcep, ... and unreliable (maxRetransmits=4294967295) channel should be created via DCEP
• dict.dict007: RTCPeerConnection-createDataChannel, createDataChannel with both maxPacketLifeTime and maxRetransmits should throw TypeError (SyntaxError changed to TypeError in the meantime)
• dict.dict008: discarded, reason: moot. Attribute check exists in historical, RTCDataChannel member maxRetransmitTime should not exist.
• dict.dict009: RTCPeerConnection-createDataChannel, createDataChannel with both maxPacketLifeTime and maxRetransmits null should succeed
• dict.dict010: RTCPeerConnection-createDataChannel, createDataChannel attribute default values and RTCDataChannel-dcep, Ordered and reliable channel should be created via DCEP
• dict.dict011: RTCDataChannel-dcep, Unordered and reliable channel should be created via DCEP
• dict.dict012: RTCPeerConnection-createDataChannel, createDataChannel attribute default values
• dict.dict013: RTCPeerConnection-ondatachannel, In-band channel created on remote peer should match the same configuration as local peer
• dict.dict014: RTCPeerConnection-ondatachannel, In-band channel created on remote peer should match the same default configuration as local peer
• dict.dict015: partially invalid (id cannot be specified without setting negotiated to true any more), RTCPeerConnection-ondatachannel, In-band channel created on remote peer should match the same configuration as local peer
• dict.dict016: discarded, reason: IDL harness tests cover readonly attributes.
• dict.dict017: RTCPeerConnection-createDataChannel, Channels created after SCTP transport is established should have id assigned and RTCPeerConnection-ondatachannel, Data channel created on remote peer should match the same default configuration as local peer
• dict.dict018: RTCPeerConnection-ondatachannel, Data channel created on remote peer should match the same default configuration as local peer
• dict.dict019: discarded, reason: IDL harness tests cover readonly attributes.
• dict.dict020: discarded, reason: id is now being ignored if negotiated is not true and that is tested in RTCPeerConnection-createDataChannel, createDataChannel with negotiated false and id 42 should ignore the id
• dict.dict021: RTCDataChannel-id, In-band negotiation with a specific ID should not be allowed
• dict.dict022: discarded, reason: negotiated channels are extensively tested in RTCDataChannel-send-receive
• dict.dict023: discarded, reason: asymmetric negotiated channels are extensively tested in RTCDataChannel-send-receive
• dict.dict024: discarded, reason: moot. Attribute check exists in historical, RTCDataChannel member reliable should not exist
• dict.dict025: discarded, reason: invalid since the spec removed the attribute
• dict.dict026: discarded, reason: invalid since the spec removed the attribute
• create.create001: RTCPeerConnection-createDataChannel, createDataChannel with closed connection should throw InvalidStateError
• create.create002: RTCPeerConnection-createDataChannel, Reusing a data channel id that is in use should throw OperationError (ResourceInUse changed to OperationError in the meantime)
• create.create003: RTCPeerConnection-createDataChannel, Reusing a data channel id that is in use (after setRemoteDescription) should throw OperationError (ResourceInUse changed to OperationError in the meantime)
• create.create004: RTCPeerConnection-createDataChannel, createDataChannel attribute default values
• create.create005: RTCDataChannel-onopen, Data channel onopen event should be fired when the data channel opens
• create.create006: RTCPeerConnection-close, Closing the local peer connection should close all channels (after connection establishment)
• create.create007: RTCPeerConnection-close, Closing the remote peer connection should close all channels (after connection establishment)
• create.create008: RTCPeerConnection-close, Closing the local peer connection should close all channels (after connection establishment)
• create.create009: RTCPeerConnection-close, Closing the remote peer connection should close all channels (after connection establishment)
• id.id001: RTCDataChannel-dcep, Channel IDs should be synchronized when created via DCEP
• id.id002: RTCPeerConnection-createDataChannel, createDataChannel with provided parameters should initialize attributes to provided values
• id.id003: discarded, reason: invalid (negotiated variant is not an edge case and none of the browsers misbehave when using it)
• id.id004: discarded, reason: invalid (negotiated variant is not an edge case and none of the browsers misbehave when using it)
• id.id005: discarded, reason: invalid (the maximum ID is 65534, a test for this exists in RTCPeerConnection-createDataChannel, createDataChannel with id 65534 should succeed)
• id.id006: RTCPeerConnection-createDataChannel, createDataChannel with id 65536 should throw TypeError
• id.id007: discarded, reason: invalid (negotiated variant is not an edge case and none of the browsers misbehave when using it)
• id.id008: discarded, reason: invalid (negotiated variant is not an edge case and none of the browsers misbehave when using it)
• id.id009: discarded, reason: invalid (browsers supporting less than 65535 streams is tested in RTCDataChannel-id, Creating a channel after exhausting the maximum number of channels should throw OperationError (after connection establishment))
• id.id010: implicitly tested in RTCDataChannel-id, Creating a channel after exhausting the maximum number of channels should throw OperationError (after connection establishment)
• id.id011: implicitly tested in RTCDataChannel-id, Exhausting channels with one peer should not violate the odd/even rule
• id.id012: discarded, reason: not an edge case (and doesn't crash Firefox any more)
• id.id013: discarded, reason: not an edge case (and doesn't provoke misbehaviour any more)
• id.id014: discarded, invalid (negotiated variant is implicitly tested in RTCDataChannel-id, Exhausting channels with one peer should not violate the odd/even rule)
• id.id015: discarded, reason: not an edge case (and doesn't crash Firefox any more)
• id.id016: discarded, reason: invalid (negotiated is not an edge case and doesn't crash Firefox any more)
• id.id017: discarded, reason: invalid (negotiated variant is in RTCDataChannel-id, ID reuse should be possible once the former channel with the same ID closed)
• id.id018: discarded, reason: test seems incomplete
• event.event001: discarded, reason: events as a result of an explicit API call on the local side should not raise events on the local side, see: Should a signalingstatechange event be fired when closing a RTCPeerConnection? w3c/webrtc-pc#1799 (yep, I'm not fond of it either). Therefore, closing on both sides is a race and can't be tested properly.
• event.event002: RTCDataChannel-close, In-band data channels ... closing by local side should be synchronized across peers
• event.event003: RTCDataChannel-close, In-band data channels ... closing by remote side should be synchronized across peers
• event.event004: Split into RTCDataChannel-send-receive, Should be able to send (local) simple string and receive (remote) as string and Should be able to send (remote) simple string and receive (local) as string
• event.event005: Split into RTCDataChannel-onopen, In-band channel: Open event should be fired (local) when the data channel opens, In-band channel: Open event should be fired (remote) when the data channel opens and Negotiated channel: Open event should be fired when the channels open
• event.event006: RTCDataChannel-close, In-band data channel closing after connection establishment
• send.send001: RTCDataChannel-send-receive, Calling send() when not open should throw InvalidStateError
• send.send002: RTCDataChannel-send-receive, ... Sending and receiving 16 KiB x64 should succeed
• send.send003: RTCDataChannel-send-receive, ... Sending after the channel has been closed (local) should raise InvalidStateError
• send.send004: RTCPeerConnection-createDataChannel, createDataChannel attribute default values
• send.send005: discarded, reason: Implicitly tested by using ArrayBuffer to receive messages, see RTCDataChannel-send-receive, ... Should be able to send Uint8Array message and receive as ArrayBuffer.
• send.send006: discarded, reason: Implicitly tested by using Blob to receive messages, see RTCDataChannel-send-receive, ... Should be able to send ArrayBuffer message and receive as Blob.
• send.send007: discarded, reason: Implicitly tested by using ArrayBuffer to receive messages and then switching to Blob, see RTCDataChannel-send-receive, ... Receiving multiple messages with different types should succeed
• send.send008: RTCDataChannel-send-receive, ... Setting binaryType should throw SyntaxError on unsupported type
• send.send009: discarded, reason: not different to the previous test (other than being an empty string).
• send.send010: RTCDataChannel-bufferedAmount, bufferedAmount initial value should be 0 for both peers
• send.send011: partially invalid (the bufferedAmount value should increase after each send call until the task returns to the event loop), RTCDataChannel-bufferedAmount, bufferedAmount should increase to byte length of encoded unicode string sent
• send.send012: discarded, reason: invalid (it's a race condition). A similar test exists in RTCDataChannel-close, bufferedAmount should not decrease immediately after initiating closure
• send.send013: discarded, reason: invalid (it's a race condition). A similar test exists in RTCDataChannel-close, bufferedAmount should not decrease after closing the peer connection
• send.send014: RTCDataChannel-send-receive, ... Should ignore binaryType and always receive string message as string
• send.send015: RTCDataChannel-send-receive, ... Should be able to send Blob message and receive as ArrayBuffer and ... Should be able to send ArrayBuffer message and receive as Blob
• send.send016: RTCDataChannel-send-receive, Should be able to send ArrayBuffer message and receive as Blob
• send.send017: RTCDataChannel-send-receive, ... Should be able to send Int32Array message and receive as ArrayBuffer
• send.send018: RTCDataChannel-send-receive, ... Should be able to send Blob message and receive as ArrayBuffer
• send.send019: RTCDataChannel-send-receive, ... Should be able to send ArrayBuffer message and receive as ArrayBuffer
• send.send020: RTCDataChannel-send-receive, ... Should be able to send Uint8Array (with offset) message and receive as ArrayBuffer
• send.send021: a similar type is being tested in RTCDataChannel-send-receive, ... Should be able to send Uint8Array (with offset) message and receive as ArrayBuffer
• send.send022: discarded, reason: I don't understand the purpose of this test (some of the array is left zero-padded... doesn't seem particularly interesting)
• send.send023: discarded, reason: I don't understand the purpose of this test (some of the array is left zero-padded... doesn't seem particularly interesting)
• send.send024: RTCDataChannel-send-receive, ... Should be able to transmit an empty string
• send.send025: RTCDataChannel-send-receive, ... Should be able to send unicode string and receive as unicode string
• send.send026: discarded, reason: spec does not explicitly state what is supposed to happen (RTCDataChannel.send(null) w3c/webrtc-pc#1835)
• send.send027: RTCDataChannel-send-receive, ... Sending and receiving 33554432 bytes should succeed or raise TypeError
• send.send028: RTCDataChannel-send-receive, ... Sending and receiving 131072 bytes should succeed or raise TypeError
• send.send029: discarded, reason: EOR not handled (when using default sized buffers of usrsctp) is already covered by ... Sending and receiving 131072 bytes should succeed or raise TypeError
• send.send030: discarded, reason: EOR not handled (when using default sized buffers of usrsctp) is already covered by ... Sending and receiving 131072 bytes should succeed or raise TypeError
• send.send031: RTCDataChannel-send-receive, ... Sending after the channel has been closed (remote) should raise InvalidStateError
• send.send032: discarded, reason: sending 32 MiB of data has already been covered (and the various typed arrays as well)
• send.send033: a similar test exists in RTCDataChannel-send-receive, ... Sending and receiving 16 KiB x64 should succeed
• send.send034: discarded, reason: spec does not explicitly state what is supposed to happen (RTCDataChannel.send(null) w3c/webrtc-pc#1835)
• send.send035: RTCDataChannel-send-receive, ... Sending and receiving 16 KiB x64 on both peer simultaneously should succeed
• send.send036: RTCDataChannel-send-receive, ... Sending and receiving 2048 messages should succeed and keep order
• send.send037: RTCDataChannel-send-receive, ... Sending and receiving 2048 messages should succeed and keep order
• send.send038: discarded, reason: doesn't fit into the scope of a WPT test (takes too long, should rather be used as a stress test)
• send.send039: RTCDataChannel-send-receive, ... Sending and receiving 262144 bytes should succeed or raise TypeError
• send.send040: discarded, reason: already covered by RTCDataChannel-send-receive, ... Sending and receiving 524288 bytes should succeed or raise TypeError
• send.send041: discarded, reason: doesn't fit into the scope of a WPT test (takes too long)
• send.send042: discarded, reason: not testable in a reliable environment

Please, let me know if you think any of the discarded tests should have been covered (especially if there are known bugs involved that I haven't covered).

Also, if you would like to keep this test suite up to date and don't want to use the WPT test suite for WebRTC, some tests will need updating since the spec has changed. I have added a note if that was the case (albeit I don't want to rule out having missed it here and there). Be aware, the WPT test suite also covers a lot of other test cases related to data channels that are not listed here.