Peer connection interop testing

[sipsorcery]

As a follow up from Sean's FOSDEM talk today I for one would be very interested in any pragmatic way to allow interop testing between different RTC Peer Connection implementations.

The library I work on has a C# implementation and I do occasionally test it with the Pion demo that uses a cut and paste of the SDP. If there was some agreed signaling approach that would encourage more frequent testing. A HTTP/REST or web socket mechanism would seem to be the most pragmatic so as to allow the same test to be run test from a browser.

[Sean-Der]

🔥 yes I would love to see this happen, I am ready to get it started.

• DataChannels could be the best way to get RPC between processes (P2P, No Server Requirement, Wide language support, Can be used for high performance)
• Building new implementations/servers will be way easier
• Learning WebRTC. Reading these tests is better then any book

@jlaine @shinyoshiaki @rainliu @metaclips I think this could be an amazing resource. Interested in getting involved/any input? I am happy to write the tests, would love your design reviews.

@steely-glint what do you think? You know the big picture better then anyone else. I would like to follow your lead if possible :) I am happy to prototype/make a feature list etc...

[rainliu]

Sure, webrtc-rs is definitely interested in getting involved in it.

[shinyoshiaki]

sounds good！

[steely-glint]

I tried to get something like this going in March last year for the IETF hackathon, but the virus got in the way. We were looking at the datachannel, rather than media.

We could look at using @murillo128 's WHIP for the signalling.

https://datatracker.ietf.org/doc/draft-murillo-whip/

It wouldn't test everything - but it would make testing the simple cases much easier - and I really want to avoid spending time in signalling issues.

|pipe| would be keen to test our stuff.

[murillo128]

Implementing an adhoc protocol over DC seems an overkill.

The the only thing is that WHIP is send only and do not cover datachannels, but nothing prevents you to do something slightly different with same SDP o/a via simple http requests.

Anyway having a list of things intended to be tested would help.

FWIW I have already a WHIP test server over Medooze Media Server available for testing.

.

[sipsorcery]

I'd be happy to use any testing approach that's agreed upon but my 2c is that I'd really like to start with the simplest possible option to test a peer connection, preferably sans the data channel requirement to start with. I've found echo tests, where the answering peer echoes back any media it gets, as the simplest option. That also seems to fit with some of the WebRTC servers like Janus and Jitsi which have an echo option built in.

If each WebRTC library had a console/browser/? application that could instantiate an echoing peer and where the SDP and ICE candidates were exchanged over a HTTP/Web Socket/? connection that seems to be like a great starting point. Building other tests, including data channels, on top of that would then have a solid foundation.

As far as something concrete goes the HTTP API I use for creating peer connections between two instances of my own library has 3 methods:

[HttpGet("{to}/{from}/{type=any}")]
[HttpPut("sdp/{from}/{to}")]
[HttpPut("ice/{from}/{to}")]

The to and from are arbitrary ID's that simply need to be opposite on each end of the peer connection. Crude javascript example here.

[sipsorcery]

Some progress to report.

I've got 3 WebRTC Echo Test servers up and running:

• My own SIPSorcery C# .NET Core application.
• A stripped down version of the Python aiortc server demo to act as an echo server.
• The Janus echo test plugin.

I'm able to do basic sanity testing with both Browsers and with a Windows Console app using my WebRTC stack.

Links for the browser based test pages are available on sipsorcery.cloud (halfway down in the WebRTC Test section) for anyone that's interested.

By getting the Echo Test servers to complete their ICE gathering before supplying an answer (the echo test is more concerned with interoperability rather than network discovery) the signaling can be reduced to a single HTTP POST. In javascript:

this.sendOffer = async function (offer) {
        return (await fetch(signalUrl, {
            method: 'POST',
            body: JSON.stringify(offer),
            headers: { 'Content-Type': 'application/json' }
        })).json();
    }

Again for those interested a POST request with a valid offer should get an SDP answer from the URL's below:

https://sipsorcery.cloud/aiortc/echo/offer
https://sipsorcery.cloud/janus/echo/offer
https://sipsorcery.cloud/sipsorcery/echo/offer

I don't know how far I'll get with further implementations but if anyone else already has a demo that's close to an Echo Test server I'd be interested to know about it. The adapted aiortc server demo is probably the most succinct Echo Test version to see what's involved.

Note that I'm not trying to set up a centralised testing server or that sort of thing. I find it very handy to have apps using other WebRTC stacks that I can run locally during development rather than only testing against Browsers.

[Sean-Der]

This is so amazing, nice work! I just submitted a PR for Pion. This is what I hope to see at webrtc-interop.org (or w/e we want to call it) A table for each feature (or test), each side will run as the offerer/answerer for one run.

Renegotation

	Google WebRTC	sipsorcery	aiortc	Pion
Google WebRTC	❌	✔️	✔️	❌
sipsorcery	✔️	❌	✔️	❌
aiortc	✔️	❌	✔️	❌
pion	❌	✔️	✔️	❌

@sipsorcery I bet we could start something really basic ASAP. We just stand up a VM and a Dockerfile that generates the HTML. We just pull from master and run our tests every 12 hours.

[shinyoshiaki]

I just submitted a PR for werift.
sipsorcery/webrtc-echoes#2

[sipsorcery]

That was quick! I'll simplify my own echo test and get it added as well.

[steely-glint]

I am confused, how can ice complete before seeing the answer ? Surely the offer side needs the ufrag from the answer?
Or are you mandating ice-lite only?

More generally this is a test of SDP - which |pipe| doesn't do. We assume that the edges will transform the SDP into some minimal JSON before the offer reaches us.

In many cases we don't even bother to exchange O/A (for an upgrade) - since we already know what the answer will be.

[jlaine]

Not directly related to WebRTC, but there is a very cool setup for testing QUIC implementations against each other. The tests are run periodically via github actions:

https://interop.seemann.io/
https://github.com/marten-seemann/quic-interop-runner

[sipsorcery]

https://github.com/sipsorcery/webrtc-echoes