Some feedback and question about send_pli

[ndrean]

I tested the ExWebRTC Echo, a ExWebRTC P-Ex-P and the WebRTC API, on OSX and Linux.

I just used send_rtc, not send_pli.
I don't use : rtcp.

What does this do or bring? Is this explaining my results?

Results

• The Echo works with Chrome (C), Firefox(F) and Safari(S), iPhone(I) but not Android(A). For the last one, if I randomly happen to display the echo, it freezes.

The error logs. Most of time, I had:

[warning] Failed to decrypt RTP, reason: :replay_fail, 

and very very rarely: 

[error] ** (Bandit.HTTPError) Header read timeout

otherwise nothing. I only notice that the transfer rate drops.

• The P_ex_P works with:
  • C <->C, but it takes up to 2 min (yes) to get the second retransmission (the first peer retransmits rather quickly the second peer)
  • C <->F : the fastest by far. 250kBps for F vs 80 for C.
  • S <-> ..., nothing, no retransmission, dashboard shows data exchange, no error in logs
  • nor with I: I never get a retransmission, dashboard shows that both peer send and receive data, no error in logs. Codec?
  • and A, it freezes rather quickly with Failed to decrypt RTP, reason: :replay_fail,

[LVala]

Hi, thanks for testing ExWebRTC!

First of all:

• send_rtp: sends the RTP packets containing the actual media,
• send_rtcp: sends RTCP packets containing metadata about the stream,
• send_pli: utility functions to make sending PLI easier,

These are quite general concepts related to media streaming so I'm not gonna go in depth here. Just google it or read the RFCs (the good starting point is the RFC 3550).

My questions:

1. Do you mean send_rtp instead of send_rtc, and by "I don't use : rtcp" do you mean that you don't send anything using the send_rtcp function?
2. Please be more specific about the scenarios you've tested in. Did you modify the Echo example? By "ExWebRTC P-Ex-P" I assume you mean PeerConnection in browser -> PeerConnection in Elixir -> PeerConnection in browser? Even if that's correct, that's too little for me to be able to deduce anything about the issues you have encountered. It would be best to just share the code.
3. How did you run the echo example on mobile? Did you try to connect from a web browser on Android/IOS to Elixir PeerConnection running on Linux/MacOS etc (if using the default HTTP, that shouldn't work at all as Chrome does not allow the use of getUserMedia API without HTTPS on non-localhost)? On mobile, did you use a mobile browser (i.e. Chrome on Android) or did you build an app and use the native APIs for WebRTC?

Without more specific info I don't know what might cause the issues, but:

• what do you mean by "retransmission"? Simply receiving the media? And by "the fastest" do you mean the lowest latency when connecting or the highest bitrate? "Retransmissions" or RTX is an actual concept in WebRTC, but it doesn't seem like that's what you mean.
• if you don't see the video (black screen of freeze) for more than a few seconds, most likely a keyframe was lost somewhere or was not produced at all and we don't forward Picture Loss Indication request in the echo example (that's what send_pli is for). It would be a common issue when connecting two peers (when the first one connects, it produces a keyframe, but there are no other peers, so the keyframe dropped. When the second peer connects, the first one does not know that it has to produce a new keyframe without using PLI, thus the long freeze),
• the bitrate is determined by the initial sender and that's always the browser or an app, Elixir WebRTC has nothing to do with that and that's not a performance issue (250 kbps is basically nothing, it should be able to handle much much more)

Generally, we have used Elixir WebRTC to connect two different PeerConnections on different platforms using the Broadcaster app and we did not encounter any critical issues.

The Failed to decrypt RTP, reason: :reply_fail is most likely irrelevant in this case, we should probably remove it to stop confusing people.

[ndrean]

Thanks for your response.

Do you mean send_rtp instead of send_rtc. Yes , typo.

• Did you modify the Echo example? Yes, I used it in a liveview. But I of course do the same, in particular defp handle_webrtc_msg. Generally speaking, all I do is follow your examples.

• Sorry for the confusion. P-EX-P means two browsers 1 and 2 create a PeerConnection and communicate with a "ExWebRTC"-PeerConnection. So I have 1 <-> ExPC1, and 2 <-> ExPC2.
  All I do is send_rtc a packet received from 1 in the "ExPC1" process to the "ExPC2" process under ExPC2 track id.
  So if you have patience to read, here:
  https://github.com/ndrean/RTC/blob/124054515ffe6156ab2ee87d0c740420e97cdf7d/lib/rtc/room_server.ex#L161
  I thought for quite a while it didn't work because I was using Safari on my computer.

• How did you run the echo example on mobile? Yes, of course. I configured Phoenix with self certificates to have an HTTPS endpoint on say 4001 because of the restriction of getUserMedia you mention. Nothing special.
  https://github.com/ndrean/RTC/blob/124054515ffe6156ab2ee87d0c740420e97cdf7d/config/dev.exs#L7
  So I open a mobile browser - Android Chrome or iPhone Chrome - to reach this (for example) "https://192.180.1.79:4001" with the app running on the mac.

• what do you mean by "retransmission"? I run the app on my mac. I open Chrome on the mac, instantiate a connection to ExWebRTC, and then do the same on my Linux computer with say Firefox, then the Chrome Mac receives quickly the video from the Firefox/Linux, but then it takes up to 2 min for Firefox/Linux to receive the video from the Chrome/Mac. More correctly, it is much faster when I use Firefox than opening Chrome on the Linux. In this configuration (whether the second computer is the same Mac or the other Linux), when I make 2 Chrome to communicate, it takes up to 2 min. With Safari<->Chrome, it simply doesn't work.

• I don't know the Retransmission concept in webRTC. I just mean that you see in one computer what the other camera of the other computer captures.

It would be a common issue when connecting two peers (when the first one connects, it produces a keyframe, but there are no other peers, so the keyframe dropped. When the second peer connects, the first one does not know that it has to produce a new keyframe without using PLI, thus the long freeze),

This is a very important notion! Should be in the doc?!

So I tried send_pli in the handle_info(:connected).
Big progress for Chrome and Firefox in terms of setup speed. Except Safari on computer and mobiles (iphone and android).
https://github.com/ndrean/RTC/blob/b04bece9c48e2ff74568fb2cd5e228d5a213f22b/lib/rtc/room_server.ex#L196
I will try to run the app from the Linux to see if there is not related to the sharing of the camera?

As for the bitrate, I excepted much more (isn't it UDP? or at least, I see in the logs that all (?) TCP based candidates "can't be unmarshal" ). The data comes from the browser peerConection.getStats(), an average of the bitsize during 2s with a setInterval.

Then I imagine all your work and demo work. I am just trying to fill in the gaps. Thanks for your help

[LVala]

Thanks, I will take a closer look tomorrow. One tip: in Chrome you can type chrome://webrtc-internals in the address bar - you'll see more or less what's in the peerConnection.getStats() but with nice plots and more or less human-readable. A bit poorer version in Firefox under about:webrtc.

[LVala]

When it comes to the bitrate, you set the resolution in mediaConstraints (at least in assets/js/webRTC.js and some other file) to 320x160 so no it's not surprising that the bitrate is low. You can change it to a higher resolution (like 1280x720) or just set to {video: true, audio: true}. The quality and the bitrate will be higher.

[LVala]

And when it comes to bitrates, everyone uses kbps (kilobits per second) and not kBps (kilobytes per second), so yeah, 200 kBps equals about 1600 kbps (~1.6 mbps) which is a normal bitrate for something like a full-hd video stream)

[LVala]

I've tested your app (the WebRTC Room with 2 peers). I was running the Phoenix app on my MacOS (M1) and tried to access the app using Chrome + Chrome, Chrome + Firefox, Chrome + Chrome on Android, Chrome on Android + Chrome on Android and everything worked fine. On Safari + Chrome I got a black screen on both, but the Chrome generated a lot of PLI requests, so this might be an issue.

You can try to implement PLI forwarding (when one peer produces PLI, forward it to the other peer) instead of just sending the PLI manually at the beginning and see if it helps. We did something similar here.

[ndrean]

Quite a newbie is this domain indeed. Thanks for the kbps vs the kBps and your explanations and your time.

So no top of forwarding PLI once both peers are connected to the server, I added the PLI forwarding in the rtcp handler as you did.

When using Chrome or Firefox, there is no forwarding triggered, everything works (Chrome Android send 1).

When I use Safari or iPhone and try to reach a Chrome or Firefox, the PLI forwarding is triggered in the rtcp event (from sides) but without any effect: black screen.

When I try to connect Safari to Safari, the first session gets the video, but the second stays black. No forwarding PLI is triggered in the rtcp event.

https://github.com/ndrean/RTC/blob/7f1b79c49f152cbc9acaa740c01e2103e3806be1/lib/rtc/room_server.ex#L181

[LVala]

Indeed, something's wrong with the Safari, but I don't have more time for this today. I'll see what I can do and get back to you.

[ndrean]

Note that WebRTC (browser) works (of course) with Safari or iPhone, as demonstrated in my third tab where I allowed up to 3 peers.